[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:02:53 dist-test-slave-g0zb krb5kdc[12800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:02:53 dist-test-slave-g0zb krb5kdc[12800](info): set up 2 sockets
Jan 09 21:02:53 dist-test-slave-g0zb krb5kdc[12800](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20250109 21:02:54.580121 12793 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:02:54 dist-test-slave-g0zb krb5kdc[12800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:02:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:02:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:02:54.715936 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:40915
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:40641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:40915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:02:54.831945 12813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:02:54.832247 12813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:02:54.832347 12813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:02:54.835779 12813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:02:54.835855 12813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:02:54.835893 12813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:02:54.835932 12813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:02:54.835976 12813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:02:54.840353 12813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:40641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:40915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:40915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:02:54.841503 12813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:02:54.842406 12813 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:02:54.850286 12818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:54.850979 12819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:54.851984 12821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:02:54.852926 12813 server_base.cc:1034] running on GCE node
I20250109 21:02:54.853494 12813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:02:54.854499 12813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:02:54.855664 12813 hybrid_clock.cc:648] HybridClock initialized: now 1736456574855635 us; error 46 us; skew 500 ppm
Jan 09 21:02:54 dist-test-slave-g0zb krb5kdc[12800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456574, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:02:54.858703 12813 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:02:54.859722 12813 webserver.cc:458] Webserver started at http://127.12.126.126:33371/ using document root <none> and password file <none>
I20250109 21:02:54.860291 12813 fs_manager.cc:362] Metadata directory not provided
I20250109 21:02:54.860348 12813 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:02:54.860536 12813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:02:54.862474 12813 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "24e550d1159e4de89705331be9dc2479"
format_stamp: "Formatted at 2025-01-09 21:02:54 on dist-test-slave-g0zb"
server_key: "e9006198b7479c5985438fab49328f3f"
server_key_iv: "8573707d156c73b76b750df690f554be"
server_key_version: "encryptionkey@0"
I20250109 21:02:54.862987 12813 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "24e550d1159e4de89705331be9dc2479"
format_stamp: "Formatted at 2025-01-09 21:02:54 on dist-test-slave-g0zb"
server_key: "e9006198b7479c5985438fab49328f3f"
server_key_iv: "8573707d156c73b76b750df690f554be"
server_key_version: "encryptionkey@0"
I20250109 21:02:54.866485 12813 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250109 21:02:54.868999 12828 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:54.870177 12813 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:02:54.870301 12813 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "24e550d1159e4de89705331be9dc2479"
format_stamp: "Formatted at 2025-01-09 21:02:54 on dist-test-slave-g0zb"
server_key: "e9006198b7479c5985438fab49328f3f"
server_key_iv: "8573707d156c73b76b750df690f554be"
server_key_version: "encryptionkey@0"
I20250109 21:02:54.870415 12813 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:02:54.886157 12813 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:02:54.889560 12813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:02:54.889751 12813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:02:54.896824 12813 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:40915
I20250109 21:02:54.897051 12880 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:40915 every 8 connection(s)
I20250109 21:02:54.897924 12813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:02:54.900597 12881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:02:54.902537 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 12813
I20250109 21:02:54.902681 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:02:54.903704 12793 external_mini_cluster.cc:1453] Setting key c32a4bb29d6db673af69a5816318a515
I20250109 21:02:54.905889 12881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479: Bootstrap starting.
I20250109 21:02:54.908646 12881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479: Neither blocks nor log segments found. Creating new log.
I20250109 21:02:54.909587 12881 log.cc:826] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479: Log is configured to *not* fsync() on all Append() calls
I20250109 21:02:54.911665 12881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479: No bootstrap required, opened a new log
I20250109 21:02:54.915521 12881 raft_consensus.cc:357] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e550d1159e4de89705331be9dc2479" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40915 } }
I20250109 21:02:54.915782 12881 raft_consensus.cc:383] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:02:54.915858 12881 raft_consensus.cc:738] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e550d1159e4de89705331be9dc2479, State: Initialized, Role: FOLLOWER
I20250109 21:02:54.916384 12881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e550d1159e4de89705331be9dc2479" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40915 } }
I20250109 21:02:54.916553 12881 raft_consensus.cc:397] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:02:54.916627 12881 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:02:54.916734 12881 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:02:54.917930 12881 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e550d1159e4de89705331be9dc2479" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40915 } }
I20250109 21:02:54.918272 12881 leader_election.cc:304] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24e550d1159e4de89705331be9dc2479; no voters:
Jan 09 21:02:54 dist-test-slave-g0zb krb5kdc[12800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:02:54.918593 12881 leader_election.cc:290] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:02:54.918685 12886 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:02:54.918874 12886 raft_consensus.cc:695] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479 [term 1 LEADER]: Becoming Leader. State: Replica: 24e550d1159e4de89705331be9dc2479, State: Running, Role: LEADER
I20250109 21:02:54.919219 12886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e550d1159e4de89705331be9dc2479" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40915 } }
I20250109 21:02:54.919827 12881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:02:54.920408 12886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24e550d1159e4de89705331be9dc2479. Latest consensus state: current_term: 1 leader_uuid: "24e550d1159e4de89705331be9dc2479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e550d1159e4de89705331be9dc2479" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40915 } } }
I20250109 21:02:54.920615 12886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479 [sys.catalog]: This master's current role is: LEADER
I20250109 21:02:54.921058 12892 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:02:54.922574 12887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24e550d1159e4de89705331be9dc2479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e550d1159e4de89705331be9dc2479" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40915 } } }
I20250109 21:02:54.922674 12887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479 [sys.catalog]: This master's current role is: LEADER
I20250109 21:02:54.924912 12892 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:02:54.925285 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:54.906035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59230 (local address 127.12.126.126:40915)
0109 21:02:54.906578 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:54.906589 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:54.906922 (+ 333us) server_negotiation.cc:407] Connection header received
0109 21:02:54.910127 (+ 3205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:54.910145 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:54.910455 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:54.910765 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:02:54.913081 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:54.913971 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:54.914992 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:54.915210 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:54.919418 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:02:54.919442 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:02:54.919452 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:02:54.919492 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:02:54.921748 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:54.922415 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:54.922424 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:54.922428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:54.922516 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:54.922890 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:54.922893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:54.922894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:54.923391 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:02:54.923628 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:54.924207 (+ 579us) server_negotiation.cc:299] Negotiation successful
0109 21:02:54.924484 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":140,"threads_started":1}
I20250109 21:02:54.930317 12892 catalog_manager.cc:1348] Generated new cluster ID: fb6a93dbf0094df8a4d7ac92a02ab3b2
I20250109 21:02:54.930481 12892 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:02:54.956665 12892 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:02:54.958029 12892 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:02:54.975152 12892 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 24e550d1159e4de89705331be9dc2479: Generated new TSK 0
I20250109 21:02:54.975944 12892 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
I20250109 21:02:55.015545 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:40915
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:40641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:02:55.127368 12907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:02:55.127631 12907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:02:55.127708 12907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:02:55.131094 12907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:02:55.131168 12907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:02:55.131284 12907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:02:55.135802 12907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:40641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:40915
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:02:55.137023 12907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:02:55.137919 12907 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:02:55.146384 12912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:55.147362 12915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:02:55.148423 12907 server_base.cc:1034] running on GCE node
W20250109 21:02:55.148876 12913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:02:55.149334 12907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:02:55.149928 12907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:02:55.151103 12907 hybrid_clock.cc:648] HybridClock initialized: now 1736456575151079 us; error 46 us; skew 500 ppm
Jan 09 21:02:55 dist-test-slave-g0zb krb5kdc[12800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456575, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:02:55.154218 12907 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:02:55.155222 12907 webserver.cc:458] Webserver started at http://127.12.126.65:37633/ using document root <none> and password file <none>
I20250109 21:02:55.155809 12907 fs_manager.cc:362] Metadata directory not provided
I20250109 21:02:55.155879 12907 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:02:55.156059 12907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:02:55.157938 12907 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "5908a15fb173468a883188da9dca26ff"
format_stamp: "Formatted at 2025-01-09 21:02:55 on dist-test-slave-g0zb"
server_key: "995188848b9f4b2038640617540d3331"
server_key_iv: "ffbe54a7105ec5323d40da9506333fdd"
server_key_version: "encryptionkey@0"
I20250109 21:02:55.158427 12907 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "5908a15fb173468a883188da9dca26ff"
format_stamp: "Formatted at 2025-01-09 21:02:55 on dist-test-slave-g0zb"
server_key: "995188848b9f4b2038640617540d3331"
server_key_iv: "ffbe54a7105ec5323d40da9506333fdd"
server_key_version: "encryptionkey@0"
I20250109 21:02:55.161968 12907 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250109 21:02:55.164289 12922 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:55.165365 12907 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:02:55.165488 12907 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "5908a15fb173468a883188da9dca26ff"
format_stamp: "Formatted at 2025-01-09 21:02:55 on dist-test-slave-g0zb"
server_key: "995188848b9f4b2038640617540d3331"
server_key_iv: "ffbe54a7105ec5323d40da9506333fdd"
server_key_version: "encryptionkey@0"
I20250109 21:02:55.165580 12907 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:02:55.187302 12907 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:02:55.190654 12907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:02:55.190883 12907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:02:55.191447 12907 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:02:55.192286 12907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:02:55.192338 12907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:55.192400 12907 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:02:55.192432 12907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:55.201757 12907 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:43669
I20250109 21:02:55.201782 13035 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:43669 every 8 connection(s)
I20250109 21:02:55.202728 12907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
Jan 09 21:02:55 dist-test-slave-g0zb krb5kdc[12800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456575, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:02:55.211912 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 12907
I20250109 21:02:55.212023 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:02:55.212334 12793 external_mini_cluster.cc:1453] Setting key b37ba2aea1b5610a124e2c3d7e27191b
I20250109 21:02:55.215648 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:55.204494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:54159 (local address 127.12.126.126:40915)
0109 21:02:55.204649 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:55.204654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:55.205477 (+ 823us) server_negotiation.cc:407] Connection header received
0109 21:02:55.206715 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:55.206718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:55.206772 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:55.206893 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:02:55.208189 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:55.208703 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:55.209288 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:55.209409 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:55.211727 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:02:55.211756 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:02:55.211763 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:02:55.211798 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:02:55.213446 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:55.214041 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:55.214044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:55.214046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:55.214099 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:55.214359 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:55.214365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:55.214370 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:55.214577 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:02:55.214688 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:55.215332 (+ 644us) server_negotiation.cc:299] Negotiation successful
0109 21:02:55.215500 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250109 21:02:55.216481 13038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:55.204796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:40915 (local address 127.12.126.65:54159)
0109 21:02:55.205326 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0109 21:02:55.205364 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0109 21:02:55.206478 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:02:55.207069 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:02:55.207077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:02:55.207486 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:02:55.208043 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:55.208056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:55.208809 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:02:55.208811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:02:55.209202 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:55.209209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:55.209384 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:55.210013 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:02:55.210040 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:02:55.211536 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:02:55.213573 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:02:55.213582 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:02:55.213595 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:02:55.213945 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:02:55.214188 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:02:55.214190 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:02:55.214192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:02:55.214278 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:02:55.214716 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:02:55.214722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:02:55.215032 (+ 310us) client_negotiation.cc:769] Sending connection context
0109 21:02:55.215248 (+ 216us) client_negotiation.cc:240] Negotiation successful
0109 21:02:55.215582 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":99,"threads_started":1}
I20250109 21:02:55.217744 13036 heartbeater.cc:346] Connected to a master server at 127.12.126.126:40915
I20250109 21:02:55.217973 13036 heartbeater.cc:463] Registering TS with master...
I20250109 21:02:55.218503 13036 heartbeater.cc:510] Master 127.12.126.126:40915 requested a full tablet report, sending...
I20250109 21:02:55.219988 12845 ts_manager.cc:194] Registered new tserver with Master: 5908a15fb173468a883188da9dca26ff (127.12.126.65:43669)
WARNING: no policy specified for kudu/127.12.126.66@KRBTEST.COM; defaulting to no policy
I20250109 21:02:55.221261 12845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:54159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
I20250109 21:02:55.243022 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:40915
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:40641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:02:55.352718 13041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:02:55.352975 13041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:02:55.353052 13041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:02:55.356333 13041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:02:55.356405 13041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:02:55.356508 13041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:02:55.361007 13041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:40641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:40915
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:02:55.362174 13041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:02:55.363087 13041 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:02:55.372036 13049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:55.372591 13046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:55.372799 13047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:02:55.373764 13041 server_base.cc:1034] running on GCE node
I20250109 21:02:55.374208 13041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:02:55.374889 13041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:02:55.376070 13041 hybrid_clock.cc:648] HybridClock initialized: now 1736456575376046 us; error 43 us; skew 500 ppm
Jan 09 21:02:55 dist-test-slave-g0zb krb5kdc[12800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456575, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:02:55.378940 13041 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:02:55.379906 13041 webserver.cc:458] Webserver started at http://127.12.126.66:39741/ using document root <none> and password file <none>
I20250109 21:02:55.380451 13041 fs_manager.cc:362] Metadata directory not provided
I20250109 21:02:55.380510 13041 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:02:55.380667 13041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:02:55.382519 13041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "d0407c93f63e4725b2c9e8e35bff8d5b"
format_stamp: "Formatted at 2025-01-09 21:02:55 on dist-test-slave-g0zb"
server_key: "125b82b3920b73f296a9fe0ea44a992c"
server_key_iv: "f2238c055c015c7ec286e0e2147f5e07"
server_key_version: "encryptionkey@0"
I20250109 21:02:55.383021 13041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "d0407c93f63e4725b2c9e8e35bff8d5b"
format_stamp: "Formatted at 2025-01-09 21:02:55 on dist-test-slave-g0zb"
server_key: "125b82b3920b73f296a9fe0ea44a992c"
server_key_iv: "f2238c055c015c7ec286e0e2147f5e07"
server_key_version: "encryptionkey@0"
I20250109 21:02:55.386464 13041 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250109 21:02:55.388860 13056 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:55.390027 13041 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:02:55.390144 13041 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "d0407c93f63e4725b2c9e8e35bff8d5b"
format_stamp: "Formatted at 2025-01-09 21:02:55 on dist-test-slave-g0zb"
server_key: "125b82b3920b73f296a9fe0ea44a992c"
server_key_iv: "f2238c055c015c7ec286e0e2147f5e07"
server_key_version: "encryptionkey@0"
I20250109 21:02:55.390278 13041 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:02:55.405652 13041 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:02:55.408993 13041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:02:55.409176 13041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:02:55.409740 13041 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:02:55.410596 13041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:02:55.410646 13041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:55.410687 13041 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:02:55.410704 13041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:55.419517 13041 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:34265
I20250109 21:02:55.419553 13169 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:34265 every 8 connection(s)
I20250109 21:02:55.420482 13041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:02:55.429212 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 13041
I20250109 21:02:55.429306 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
Jan 09 21:02:55 dist-test-slave-g0zb krb5kdc[12800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456575, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:02:55.429543 12793 external_mini_cluster.cc:1453] Setting key 3871a899b82159d8bc83d4248e60b306
I20250109 21:02:55.433357 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:55.422314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:50889 (local address 127.12.126.126:40915)
0109 21:02:55.422506 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:55.422511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:55.423236 (+ 725us) server_negotiation.cc:407] Connection header received
0109 21:02:55.424494 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:55.424497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:55.424550 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:55.424634 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:02:55.426339 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:55.426896 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:55.427542 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:55.427723 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:55.429984 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:02:55.430010 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:02:55.430013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:02:55.430041 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:02:55.431499 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:55.431991 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:55.431995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:55.431996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:55.432040 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:55.432308 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:55.432311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:55.432313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:55.432467 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:02:55.432564 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:55.433061 (+ 497us) server_negotiation.cc:299] Negotiation successful
0109 21:02:55.433181 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20250109 21:02:55.434031 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:55.422593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:40915 (local address 127.12.126.66:50889)
0109 21:02:55.423089 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0109 21:02:55.423127 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0109 21:02:55.424240 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:02:55.424863 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:02:55.424875 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:02:55.425500 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:02:55.426168 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:55.426183 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:55.427023 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:02:55.427027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:02:55.427424 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:55.427432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:55.427654 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:55.428294 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:02:55.428321 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:02:55.429821 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:02:55.431621 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:02:55.431627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:02:55.431641 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:02:55.431893 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:02:55.432123 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:02:55.432126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:02:55.432128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:02:55.432217 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:02:55.432573 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:02:55.432582 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:02:55.432828 (+ 246us) client_negotiation.cc:769] Sending connection context
0109 21:02:55.433014 (+ 186us) client_negotiation.cc:240] Negotiation successful
0109 21:02:55.433312 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":95,"threads_started":1}
I20250109 21:02:55.435529 13170 heartbeater.cc:346] Connected to a master server at 127.12.126.126:40915
I20250109 21:02:55.435763 13170 heartbeater.cc:463] Registering TS with master...
I20250109 21:02:55.436297 13170 heartbeater.cc:510] Master 127.12.126.126:40915 requested a full tablet report, sending...
I20250109 21:02:55.437471 12845 ts_manager.cc:194] Registered new tserver with Master: d0407c93f63e4725b2c9e8e35bff8d5b (127.12.126.66:34265)
I20250109 21:02:55.438324 12845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:50889
WARNING: no policy specified for kudu/127.12.126.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
I20250109 21:02:55.459362 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:40915
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:40641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:02:55.575783 13175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:02:55.576035 13175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:02:55.576095 13175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:02:55.579342 13175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:02:55.579417 13175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:02:55.579512 13175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:02:55.584187 13175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:40641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:40915
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:02:55.585372 13175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:02:55.586259 13175 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:02:55.595278 13183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:55.595906 13180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:55.596046 13181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:02:55.597069 13175 server_base.cc:1034] running on GCE node
I20250109 21:02:55.597541 13175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:02:55.598169 13175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:02:55.599318 13175 hybrid_clock.cc:648] HybridClock initialized: now 1736456575599304 us; error 34 us; skew 500 ppm
Jan 09 21:02:55 dist-test-slave-g0zb krb5kdc[12800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456575, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:02:55.602253 13175 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:02:55.603266 13175 webserver.cc:458] Webserver started at http://127.12.126.67:46535/ using document root <none> and password file <none>
I20250109 21:02:55.603848 13175 fs_manager.cc:362] Metadata directory not provided
I20250109 21:02:55.603911 13175 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:02:55.604090 13175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:02:55.605959 13175 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "bb7a164edd5146ccba5fe43d4741a274"
format_stamp: "Formatted at 2025-01-09 21:02:55 on dist-test-slave-g0zb"
server_key: "3a664f45a99a71048f298d2d7f524add"
server_key_iv: "7046c14706e349b1a9ffb5c09aa7ea43"
server_key_version: "encryptionkey@0"
I20250109 21:02:55.606459 13175 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "bb7a164edd5146ccba5fe43d4741a274"
format_stamp: "Formatted at 2025-01-09 21:02:55 on dist-test-slave-g0zb"
server_key: "3a664f45a99a71048f298d2d7f524add"
server_key_iv: "7046c14706e349b1a9ffb5c09aa7ea43"
server_key_version: "encryptionkey@0"
I20250109 21:02:55.610015 13175 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250109 21:02:55.612480 13190 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:55.613583 13175 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:02:55.613708 13175 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "bb7a164edd5146ccba5fe43d4741a274"
format_stamp: "Formatted at 2025-01-09 21:02:55 on dist-test-slave-g0zb"
server_key: "3a664f45a99a71048f298d2d7f524add"
server_key_iv: "7046c14706e349b1a9ffb5c09aa7ea43"
server_key_version: "encryptionkey@0"
I20250109 21:02:55.613812 13175 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:02:55.624897 13175 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:02:55.628082 13175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:02:55.628271 13175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:02:55.628813 13175 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:02:55.629629 13175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:02:55.629678 13175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:55.629735 13175 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:02:55.629768 13175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:55.638299 13175 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:45405
I20250109 21:02:55.638334 13303 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:45405 every 8 connection(s)
I20250109 21:02:55.639293 13175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:02:55.645900 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 13175
I20250109 21:02:55.646008 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:02:55.646245 12793 external_mini_cluster.cc:1453] Setting key 104c656f83b05b2ea503a707557860f7
Jan 09 21:02:55 dist-test-slave-g0zb krb5kdc[12800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456575, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:02:55.652662 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:55.641097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:40189 (local address 127.12.126.126:40915)
0109 21:02:55.641257 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:55.641262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:55.641979 (+ 717us) server_negotiation.cc:407] Connection header received
0109 21:02:55.643335 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:55.643338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:55.643423 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:55.643508 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:02:55.644797 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:55.645345 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:55.646066 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:55.646259 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:55.648908 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:02:55.648925 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:02:55.648928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:02:55.648958 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:02:55.650626 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:55.651196 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:55.651199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:55.651200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:55.651251 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:55.651611 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:55.651614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:55.651615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:55.651797 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:02:55.651877 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:55.652412 (+ 535us) server_negotiation.cc:299] Negotiation successful
0109 21:02:55.652533 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250109 21:02:55.653312 13306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:55.641399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:40915 (local address 127.12.126.67:40189)
0109 21:02:55.641837 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0109 21:02:55.641873 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0109 21:02:55.643099 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:02:55.643651 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:02:55.643659 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:02:55.644051 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:02:55.644633 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:55.644647 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:55.645520 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:02:55.645524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:02:55.645925 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:55.645933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:55.646184 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:55.646946 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:02:55.646971 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:02:55.648734 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:02:55.650759 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:02:55.650764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:02:55.650774 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:02:55.651083 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:02:55.651366 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:02:55.651369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:02:55.651371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:02:55.651489 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:02:55.651921 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:02:55.651926 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:02:55.652179 (+ 253us) client_negotiation.cc:769] Sending connection context
0109 21:02:55.652354 (+ 175us) client_negotiation.cc:240] Negotiation successful
0109 21:02:55.652605 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":91,"threads_started":1}
I20250109 21:02:55.654470 13304 heartbeater.cc:346] Connected to a master server at 127.12.126.126:40915
I20250109 21:02:55.654716 13304 heartbeater.cc:463] Registering TS with master...
I20250109 21:02:55.655292 13304 heartbeater.cc:510] Master 127.12.126.126:40915 requested a full tablet report, sending...
I20250109 21:02:55.656373 12845 ts_manager.cc:194] Registered new tserver with Master: bb7a164edd5146ccba5fe43d4741a274 (127.12.126.67:45405)
I20250109 21:02:55.656924 12845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:40189
I20250109 21:02:55.661237 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
Jan 09 21:02:55 dist-test-slave-g0zb krb5kdc[12800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456575, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 09 21:02:55 dist-test-slave-g0zb krb5kdc[12800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456575, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.126.65@KRBTEST.COM
I20250109 21:02:55.685956 13311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:55.676553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.12.126.65:43669)
0109 21:02:55.676822 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:55.676828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:55.676853 (+ 25us) server_negotiation.cc:407] Connection header received
0109 21:02:55.676974 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:55.676979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:55.677164 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:55.677294 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:02:55.678055 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:55.678825 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:55.679524 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:55.679788 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:55.681994 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:02:55.682020 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:02:55.682033 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:02:55.682061 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:02:55.683987 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:55.684508 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:55.684519 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:55.684521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:55.684593 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:55.684882 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:55.684885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:55.684886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:55.685102 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:02:55.685304 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:55.685598 (+ 294us) server_negotiation.cc:299] Negotiation successful
0109 21:02:55.685772 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
W20250109 21:02:55.686936 12950 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59916
Jan 09 21:02:55 dist-test-slave-g0zb krb5kdc[12800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 09 21:02:55 dist-test-slave-g0zb krb5kdc[12800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.65@KRBTEST.COM
I20250109 21:02:55.711154 13311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:55.703493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.12.126.65:43669)
0109 21:02:55.703662 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:55.703668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:55.703749 (+ 81us) server_negotiation.cc:407] Connection header received
0109 21:02:55.703914 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:55.703917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:55.703969 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:55.704052 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:02:55.704883 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:55.705399 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:55.706043 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:55.706277 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:55.708402 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:02:55.708425 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:02:55.708428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:02:55.708454 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:02:55.709782 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:55.710152 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:55.710154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:55.710156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:55.710218 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:55.710429 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:55.710433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:55.710434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:55.710577 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:02:55.710702 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:55.710889 (+ 187us) server_negotiation.cc:299] Negotiation successful
0109 21:02:55.711001 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20250109 21:02:55.712174 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 12907
I20250109 21:02:55.718636 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 13041
I20250109 21:02:55.727025 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 13175
I20250109 21:02:55.733145 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 12813
2025-01-09T21:02:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2218 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:02:55 dist-test-slave-g0zb krb5kdc[13318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:02:55 dist-test-slave-g0zb krb5kdc[13318](info): set up 2 sockets
Jan 09 21:02:55 dist-test-slave-g0zb krb5kdc[13318](info): commencing operation
krb5kdc: starting...
W20250109 21:02:56.767102 12793 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:02:56 dist-test-slave-g0zb krb5kdc[13318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:02:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:02:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:02:56.889927 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:42889
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:35979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:42889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:02:57.001936 13331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:02:57.002249 13331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:02:57.002333 13331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:02:57.005538 13331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:02:57.005614 13331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:02:57.005651 13331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:02:57.005685 13331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:02:57.005725 13331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:02:57.010073 13331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:35979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:42889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:42889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:02:57.011217 13331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:02:57.012063 13331 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:02:57.019737 13339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:57.020179 13337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:57.021037 13336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:02:57.021140 13331 server_base.cc:1034] running on GCE node
I20250109 21:02:57.021648 13331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:02:57.022631 13331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:02:57.023775 13331 hybrid_clock.cc:648] HybridClock initialized: now 1736456577023765 us; error 59 us; skew 500 ppm
Jan 09 21:02:57 dist-test-slave-g0zb krb5kdc[13318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456577, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:02:57.026516 13331 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:02:57.027446 13331 webserver.cc:458] Webserver started at http://127.12.126.126:39949/ using document root <none> and password file <none>
I20250109 21:02:57.027989 13331 fs_manager.cc:362] Metadata directory not provided
I20250109 21:02:57.028050 13331 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:02:57.028224 13331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:02:57.030030 13331 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "091684f08e2343879fc769cddddb2529"
format_stamp: "Formatted at 2025-01-09 21:02:57 on dist-test-slave-g0zb"
server_key: "9d85fb01305d818c5072f7072c5dbc34"
server_key_iv: "9288dfe0cea29d40020c86acc3b26fb8"
server_key_version: "encryptionkey@0"
I20250109 21:02:57.030512 13331 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "091684f08e2343879fc769cddddb2529"
format_stamp: "Formatted at 2025-01-09 21:02:57 on dist-test-slave-g0zb"
server_key: "9d85fb01305d818c5072f7072c5dbc34"
server_key_iv: "9288dfe0cea29d40020c86acc3b26fb8"
server_key_version: "encryptionkey@0"
I20250109 21:02:57.033892 13331 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:02:57.036213 13346 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:57.037269 13331 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:02:57.037382 13331 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "091684f08e2343879fc769cddddb2529"
format_stamp: "Formatted at 2025-01-09 21:02:57 on dist-test-slave-g0zb"
server_key: "9d85fb01305d818c5072f7072c5dbc34"
server_key_iv: "9288dfe0cea29d40020c86acc3b26fb8"
server_key_version: "encryptionkey@0"
I20250109 21:02:57.037482 13331 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:02:57.048437 13331 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:02:57.051278 13331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:02:57.051448 13331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:02:57.057827 13331 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:42889
I20250109 21:02:57.058060 13398 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:42889 every 8 connection(s)
I20250109 21:02:57.058841 13331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:02:57.061290 13399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:02:57.066192 13399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529: Bootstrap starting.
I20250109 21:02:57.066622 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 13331
I20250109 21:02:57.066727 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:02:57.066967 12793 external_mini_cluster.cc:1453] Setting key b7afd12b1a77aba67a58dd2d0677961e
I20250109 21:02:57.068665 13399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529: Neither blocks nor log segments found. Creating new log.
I20250109 21:02:57.069350 13399 log.cc:826] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529: Log is configured to *not* fsync() on all Append() calls
I20250109 21:02:57.071283 13399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529: No bootstrap required, opened a new log
Jan 09 21:02:57 dist-test-slave-g0zb krb5kdc[13318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:02:57.075194 13399 raft_consensus.cc:357] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091684f08e2343879fc769cddddb2529" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42889 } }
I20250109 21:02:57.075409 13399 raft_consensus.cc:383] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:02:57.075481 13399 raft_consensus.cc:738] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 091684f08e2343879fc769cddddb2529, State: Initialized, Role: FOLLOWER
I20250109 21:02:57.075922 13399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091684f08e2343879fc769cddddb2529" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42889 } }
I20250109 21:02:57.076041 13399 raft_consensus.cc:397] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:02:57.076107 13399 raft_consensus.cc:491] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:02:57.076223 13399 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:02:57.077010 13399 raft_consensus.cc:513] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091684f08e2343879fc769cddddb2529" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42889 } }
I20250109 21:02:57.077420 13399 leader_election.cc:304] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 091684f08e2343879fc769cddddb2529; no voters:
I20250109 21:02:57.077739 13399 leader_election.cc:290] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:02:57.077816 13404 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:02:57.078033 13404 raft_consensus.cc:695] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529 [term 1 LEADER]: Becoming Leader. State: Replica: 091684f08e2343879fc769cddddb2529, State: Running, Role: LEADER
I20250109 21:02:57.078351 13404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091684f08e2343879fc769cddddb2529" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42889 } }
I20250109 21:02:57.079069 13399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:02:57.079969 13406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 091684f08e2343879fc769cddddb2529. Latest consensus state: current_term: 1 leader_uuid: "091684f08e2343879fc769cddddb2529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091684f08e2343879fc769cddddb2529" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42889 } } }
I20250109 21:02:57.080102 13406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529 [sys.catalog]: This master's current role is: LEADER
I20250109 21:02:57.080241 13405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "091684f08e2343879fc769cddddb2529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091684f08e2343879fc769cddddb2529" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42889 } } }
I20250109 21:02:57.080336 13405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529 [sys.catalog]: This master's current role is: LEADER
I20250109 21:02:57.080102 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:57.068062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.12.126.126:42889)
0109 21:02:57.068438 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:57.068447 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:57.068481 (+ 34us) server_negotiation.cc:407] Connection header received
0109 21:02:57.069038 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:57.069058 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:57.069361 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:57.069680 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:02:57.070579 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.071412 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:57.072247 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.072436 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:57.074931 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:02:57.074953 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:02:57.074967 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:02:57.075006 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:02:57.076945 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:57.077405 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:57.077410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:57.077415 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:57.077491 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:57.077861 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:57.077864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:57.077867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:57.078351 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:02:57.078632 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:57.078989 (+ 357us) server_negotiation.cc:299] Negotiation successful
0109 21:02:57.079258 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":108,"threads_started":1}
I20250109 21:02:57.080624 13413 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:02:57.084033 13413 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:02:57.089053 13413 catalog_manager.cc:1348] Generated new cluster ID: 99c71683637f41758f940c584416c265
I20250109 21:02:57.089128 13413 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:02:57.119298 13413 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:02:57.120363 13413 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:02:57.125895 13413 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 091684f08e2343879fc769cddddb2529: Generated new TSK 0
I20250109 21:02:57.126588 13413 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
I20250109 21:02:57.159052 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42889
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:35979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:02:57.267168 13425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:02:57.267410 13425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:02:57.267472 13425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:02:57.270495 13425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:02:57.270550 13425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:02:57.270627 13425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:02:57.274788 13425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:35979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42889
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:02:57.275803 13425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:02:57.276603 13425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:02:57.284235 13433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:57.285048 13430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:57.285894 13431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:02:57.286715 13425 server_base.cc:1034] running on GCE node
I20250109 21:02:57.287120 13425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:02:57.287676 13425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:02:57.288818 13425 hybrid_clock.cc:648] HybridClock initialized: now 1736456577288795 us; error 46 us; skew 500 ppm
Jan 09 21:02:57 dist-test-slave-g0zb krb5kdc[13318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456577, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:02:57.291484 13425 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:02:57.292366 13425 webserver.cc:458] Webserver started at http://127.12.126.65:32987/ using document root <none> and password file <none>
I20250109 21:02:57.292896 13425 fs_manager.cc:362] Metadata directory not provided
I20250109 21:02:57.292948 13425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:02:57.293095 13425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:02:57.294950 13425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "84552845e3b34c45ad6e0018326ab883"
format_stamp: "Formatted at 2025-01-09 21:02:57 on dist-test-slave-g0zb"
server_key: "e36e0a40cbf40c9bde92cc9e37adde87"
server_key_iv: "305a6546dbaf3ae310970e9945f4a87c"
server_key_version: "encryptionkey@0"
I20250109 21:02:57.295398 13425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "84552845e3b34c45ad6e0018326ab883"
format_stamp: "Formatted at 2025-01-09 21:02:57 on dist-test-slave-g0zb"
server_key: "e36e0a40cbf40c9bde92cc9e37adde87"
server_key_iv: "305a6546dbaf3ae310970e9945f4a87c"
server_key_version: "encryptionkey@0"
I20250109 21:02:57.298655 13425 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:02:57.300794 13440 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:57.301775 13425 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:02:57.301869 13425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "84552845e3b34c45ad6e0018326ab883"
format_stamp: "Formatted at 2025-01-09 21:02:57 on dist-test-slave-g0zb"
server_key: "e36e0a40cbf40c9bde92cc9e37adde87"
server_key_iv: "305a6546dbaf3ae310970e9945f4a87c"
server_key_version: "encryptionkey@0"
I20250109 21:02:57.301946 13425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:02:57.323637 13425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:02:57.326762 13425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:02:57.326934 13425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:02:57.327427 13425 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:02:57.328187 13425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:02:57.328243 13425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:57.328289 13425 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:02:57.328310 13425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:57.336725 13425 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:42921
I20250109 21:02:57.336776 13553 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:42921 every 8 connection(s)
I20250109 21:02:57.337669 13425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:02:57.345042 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 13425
I20250109 21:02:57.345168 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:02:57.345408 12793 external_mini_cluster.cc:1453] Setting key c944206ae1de26b1f4b8e6b41d87f4ad
Jan 09 21:02:57 dist-test-slave-g0zb krb5kdc[13318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456577, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:02:57.350972 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:57.339450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:51093 (local address 127.12.126.126:42889)
0109 21:02:57.339612 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:57.339616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:57.340292 (+ 676us) server_negotiation.cc:407] Connection header received
0109 21:02:57.341786 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:57.341789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:57.341849 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:57.341937 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:02:57.343298 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.343801 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:57.344445 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.344624 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:57.347108 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:02:57.347126 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:02:57.347128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:02:57.347155 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:02:57.348903 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:57.349436 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:57.349443 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:57.349447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:57.349512 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:57.349771 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:57.349777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:57.349781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:57.349986 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:02:57.350093 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:57.350576 (+ 483us) server_negotiation.cc:299] Negotiation successful
0109 21:02:57.350736 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20250109 21:02:57.351648 13556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:57.339724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42889 (local address 127.12.126.65:51093)
0109 21:02:57.340130 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0109 21:02:57.340193 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0109 21:02:57.341569 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:02:57.342103 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:02:57.342111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:02:57.342522 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:02:57.343157 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:57.343172 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.343925 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:02:57.343928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:02:57.344328 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:57.344336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.344566 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:57.345316 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:02:57.345341 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:02:57.346953 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:02:57.349029 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:02:57.349033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:02:57.349043 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:02:57.349301 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:02:57.349609 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:02:57.349611 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:02:57.349612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:02:57.349691 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:02:57.350078 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:02:57.350082 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:02:57.350321 (+ 239us) client_negotiation.cc:769] Sending connection context
0109 21:02:57.350508 (+ 187us) client_negotiation.cc:240] Negotiation successful
0109 21:02:57.350753 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":100,"threads_started":1}
I20250109 21:02:57.352840 13554 heartbeater.cc:346] Connected to a master server at 127.12.126.126:42889
I20250109 21:02:57.353068 13554 heartbeater.cc:463] Registering TS with master...
I20250109 21:02:57.353581 13554 heartbeater.cc:510] Master 127.12.126.126:42889 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.126.66@KRBTEST.COM; defaulting to no policy
I20250109 21:02:57.355088 13363 ts_manager.cc:194] Registered new tserver with Master: 84552845e3b34c45ad6e0018326ab883 (127.12.126.65:42921)
I20250109 21:02:57.356259 13363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:51093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
I20250109 21:02:57.373927 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42889
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:35979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:02:57.484038 13559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:02:57.484263 13559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:02:57.484325 13559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:02:57.487411 13559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:02:57.487468 13559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:02:57.487545 13559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:02:57.491794 13559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:35979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42889
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:02:57.492882 13559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:02:57.493709 13559 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:02:57.502245 13565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:57.502367 13567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:57.503120 13564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:02:57.503858 13559 server_base.cc:1034] running on GCE node
I20250109 21:02:57.504222 13559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:02:57.504765 13559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:02:57.505935 13559 hybrid_clock.cc:648] HybridClock initialized: now 1736456577505921 us; error 24 us; skew 500 ppm
Jan 09 21:02:57 dist-test-slave-g0zb krb5kdc[13318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456577, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:02:57.508723 13559 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:02:57.509737 13559 webserver.cc:458] Webserver started at http://127.12.126.66:34415/ using document root <none> and password file <none>
I20250109 21:02:57.510334 13559 fs_manager.cc:362] Metadata directory not provided
I20250109 21:02:57.510396 13559 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:02:57.510573 13559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:02:57.512482 13559 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "a39f5deee7db432fab6cb589c0b2bf49"
format_stamp: "Formatted at 2025-01-09 21:02:57 on dist-test-slave-g0zb"
server_key: "6cf734699ac8265e596b21eab6e648f8"
server_key_iv: "a4241dde670bc5bcaaedcfcc455c692c"
server_key_version: "encryptionkey@0"
I20250109 21:02:57.512975 13559 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "a39f5deee7db432fab6cb589c0b2bf49"
format_stamp: "Formatted at 2025-01-09 21:02:57 on dist-test-slave-g0zb"
server_key: "6cf734699ac8265e596b21eab6e648f8"
server_key_iv: "a4241dde670bc5bcaaedcfcc455c692c"
server_key_version: "encryptionkey@0"
I20250109 21:02:57.516559 13559 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:02:57.518787 13574 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:57.519876 13559 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:02:57.519992 13559 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "a39f5deee7db432fab6cb589c0b2bf49"
format_stamp: "Formatted at 2025-01-09 21:02:57 on dist-test-slave-g0zb"
server_key: "6cf734699ac8265e596b21eab6e648f8"
server_key_iv: "a4241dde670bc5bcaaedcfcc455c692c"
server_key_version: "encryptionkey@0"
I20250109 21:02:57.520084 13559 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:02:57.550089 13559 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:02:57.552999 13559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:02:57.553174 13559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:02:57.553779 13559 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:02:57.554576 13559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:02:57.554625 13559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:57.554683 13559 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:02:57.554715 13559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:57.563073 13559 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:45965
I20250109 21:02:57.563100 13687 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:45965 every 8 connection(s)
I20250109 21:02:57.564003 13559 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:02:57.570290 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 13559
I20250109 21:02:57.570405 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:02:57.570644 12793 external_mini_cluster.cc:1453] Setting key 46dd1e43b0e20c7473410bc09ccc62d2
Jan 09 21:02:57 dist-test-slave-g0zb krb5kdc[13318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456577, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:02:57.576037 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:57.565646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:53487 (local address 127.12.126.126:42889)
0109 21:02:57.565792 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:57.565797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:57.566456 (+ 659us) server_negotiation.cc:407] Connection header received
0109 21:02:57.567687 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:57.567690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:57.567742 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:57.567827 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:02:57.569021 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.569527 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:57.570147 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.570357 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:57.572683 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:02:57.572697 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:02:57.572700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:02:57.572722 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:02:57.574188 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:57.574653 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:57.574657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:57.574658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:57.574699 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:57.574986 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:57.574991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:57.574993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:57.575171 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:02:57.575274 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:57.575756 (+ 482us) server_negotiation.cc:299] Negotiation successful
0109 21:02:57.575875 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250109 21:02:57.576660 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:57.565921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42889 (local address 127.12.126.66:53487)
0109 21:02:57.566322 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0109 21:02:57.566356 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0109 21:02:57.567463 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:02:57.567969 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:02:57.567977 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:02:57.568345 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:02:57.568902 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:57.568914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.569620 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:02:57.569622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:02:57.570025 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:57.570032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.570230 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:57.570892 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:02:57.570916 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:02:57.572537 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:02:57.574303 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:02:57.574308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:02:57.574320 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:02:57.574558 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:02:57.574778 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:02:57.574782 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:02:57.574784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:02:57.574896 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:02:57.575282 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:02:57.575291 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:02:57.575522 (+ 231us) client_negotiation.cc:769] Sending connection context
0109 21:02:57.575694 (+ 172us) client_negotiation.cc:240] Negotiation successful
0109 21:02:57.575964 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":91,"threads_started":1}
I20250109 21:02:57.577684 13688 heartbeater.cc:346] Connected to a master server at 127.12.126.126:42889
I20250109 21:02:57.577950 13688 heartbeater.cc:463] Registering TS with master...
I20250109 21:02:57.578627 13688 heartbeater.cc:510] Master 127.12.126.126:42889 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.126.67@KRBTEST.COM; defaulting to no policy
I20250109 21:02:57.579870 13363 ts_manager.cc:194] Registered new tserver with Master: a39f5deee7db432fab6cb589c0b2bf49 (127.12.126.66:45965)
I20250109 21:02:57.580565 13363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:53487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
I20250109 21:02:57.598565 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42889
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:35979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:02:57.705929 13693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:02:57.706158 13693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:02:57.706228 13693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:02:57.709444 13693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:02:57.709514 13693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:02:57.709625 13693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:02:57.713887 13693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:35979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42889
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:02:57.714952 13693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:02:57.715793 13693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:02:57.724251 13701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:57.724298 13698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:57.725067 13699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:02:57.725900 13693 server_base.cc:1034] running on GCE node
I20250109 21:02:57.726310 13693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:02:57.726934 13693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:02:57.728082 13693 hybrid_clock.cc:648] HybridClock initialized: now 1736456577728064 us; error 35 us; skew 500 ppm
Jan 09 21:02:57 dist-test-slave-g0zb krb5kdc[13318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456577, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:02:57.730794 13693 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:02:57.731775 13693 webserver.cc:458] Webserver started at http://127.12.126.67:39939/ using document root <none> and password file <none>
I20250109 21:02:57.732345 13693 fs_manager.cc:362] Metadata directory not provided
I20250109 21:02:57.732404 13693 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:02:57.732576 13693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:02:57.734464 13693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "8365b1ef67ad41acaa7b4821938f644a"
format_stamp: "Formatted at 2025-01-09 21:02:57 on dist-test-slave-g0zb"
server_key: "1e3013101d30582aab93efc508567461"
server_key_iv: "b44cfa2c3810b49c510d8221195311aa"
server_key_version: "encryptionkey@0"
I20250109 21:02:57.734982 13693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "8365b1ef67ad41acaa7b4821938f644a"
format_stamp: "Formatted at 2025-01-09 21:02:57 on dist-test-slave-g0zb"
server_key: "1e3013101d30582aab93efc508567461"
server_key_iv: "b44cfa2c3810b49c510d8221195311aa"
server_key_version: "encryptionkey@0"
I20250109 21:02:57.738500 13693 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:02:57.740826 13708 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:57.742039 13693 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:02:57.742174 13693 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "8365b1ef67ad41acaa7b4821938f644a"
format_stamp: "Formatted at 2025-01-09 21:02:57 on dist-test-slave-g0zb"
server_key: "1e3013101d30582aab93efc508567461"
server_key_iv: "b44cfa2c3810b49c510d8221195311aa"
server_key_version: "encryptionkey@0"
I20250109 21:02:57.742280 13693 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:02:57.766566 13693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:02:57.769476 13693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:02:57.769652 13693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:02:57.770179 13693 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:02:57.771005 13693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:02:57.771056 13693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:57.771116 13693 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:02:57.771142 13693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:57.780442 13693 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:35893
I20250109 21:02:57.780473 13821 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:35893 every 8 connection(s)
I20250109 21:02:57.781468 13693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:02:57.784473 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 13693
I20250109 21:02:57.784579 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:02:57.784819 12793 external_mini_cluster.cc:1453] Setting key 341a393a371a720081b9c5ef227c5e4b
Jan 09 21:02:57 dist-test-slave-g0zb krb5kdc[13318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456577, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:02:57.793401 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:57.783198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:58101 (local address 127.12.126.126:42889)
0109 21:02:57.783353 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:57.783358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:57.784079 (+ 721us) server_negotiation.cc:407] Connection header received
0109 21:02:57.785391 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:57.785395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:57.785459 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:57.785565 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:02:57.787112 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.787624 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:57.788187 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.788301 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:57.790393 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:02:57.790409 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:02:57.790411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:02:57.790437 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:02:57.791870 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:57.792273 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:57.792276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:57.792277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:57.792313 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:57.792530 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:57.792532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:57.792533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:57.792697 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:02:57.792779 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:57.793179 (+ 400us) server_negotiation.cc:299] Negotiation successful
0109 21:02:57.793277 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250109 21:02:57.794037 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:57.783532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42889 (local address 127.12.126.67:58101)
0109 21:02:57.783942 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0109 21:02:57.783976 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0109 21:02:57.785155 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:02:57.785750 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:02:57.785761 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:02:57.786293 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:02:57.786987 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:57.787003 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.787716 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:02:57.787719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:02:57.788107 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:57.788113 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.788280 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:57.788837 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:02:57.788858 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:02:57.790254 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:02:57.791966 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:02:57.791970 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:02:57.791978 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:02:57.792198 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:02:57.792375 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:02:57.792377 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:02:57.792379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:02:57.792449 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:02:57.792775 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:02:57.792780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:02:57.792991 (+ 211us) client_negotiation.cc:769] Sending connection context
0109 21:02:57.793151 (+ 160us) client_negotiation.cc:240] Negotiation successful
0109 21:02:57.793380 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":99,"threads_started":1}
I20250109 21:02:57.795038 13822 heartbeater.cc:346] Connected to a master server at 127.12.126.126:42889
I20250109 21:02:57.795269 13822 heartbeater.cc:463] Registering TS with master...
I20250109 21:02:57.795758 13822 heartbeater.cc:510] Master 127.12.126.126:42889 requested a full tablet report, sending...
I20250109 21:02:57.796730 13363 ts_manager.cc:194] Registered new tserver with Master: 8365b1ef67ad41acaa7b4821938f644a (127.12.126.67:35893)
I20250109 21:02:57.797356 13363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:58101
I20250109 21:02:57.798612 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 21:02:57.807948 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:57.801367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.12.126.126:42889)
0109 21:02:57.801493 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:57.801497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:57.801656 (+ 159us) server_negotiation.cc:407] Connection header received
0109 21:02:57.801786 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:57.801791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:57.801832 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:57.801905 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:02:57.802733 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.803276 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:57.803956 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.804167 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:57.805220 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:02:57.805249 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:02:57.805252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:02:57.805276 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:02:57.806453 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:57.806869 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:57.806873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:57.806875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:57.806926 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:57.807135 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:57.807138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:57.807140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:57.807323 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:02:57.807448 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:57.807666 (+ 218us) server_negotiation.cc:299] Negotiation successful
0109 21:02:57.807784 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20250109 21:02:57.812045 13363 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:02:57.814572 13363 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:02:57.827059 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:57.822603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:35893 (local address 127.0.0.1:55056)
0109 21:02:57.823391 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0109 21:02:57.823432 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0109 21:02:57.823583 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:02:57.824343 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:02:57.824349 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:02:57.824414 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:02:57.824734 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:57.824741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.825761 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:02:57.825767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:02:57.826511 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:57.826521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.826629 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:57.826687 (+ 58us) client_negotiation.cc:769] Sending connection context
0109 21:02:57.826787 (+ 100us) client_negotiation.cc:240] Negotiation successful
0109 21:02:57.826909 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"spinlock_wait_cycles":64256,"thread_start_us":104,"threads_started":1}
I20250109 21:02:57.827687 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:57.823007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.12.126.67:35893)
0109 21:02:57.823312 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:57.823318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:57.823741 (+ 423us) server_negotiation.cc:407] Connection header received
0109 21:02:57.823837 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:57.823842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:57.824026 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:57.824179 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:02:57.824939 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.825642 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:57.826669 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.827241 (+ 572us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:57.827373 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:57.827450 (+ 77us) server_negotiation.cc:299] Negotiation successful
0109 21:02:57.827537 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":91,"threads_started":1}
I20250109 21:02:57.828091 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:57.823315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:45965 (local address 127.0.0.1:51396)
0109 21:02:57.823603 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0109 21:02:57.823615 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:02:57.823686 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:02:57.824583 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:02:57.824586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:02:57.824602 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:02:57.824850 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:57.824860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.826195 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:02:57.826198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:02:57.827733 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:57.827740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.827836 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:57.827855 (+ 19us) client_negotiation.cc:769] Sending connection context
0109 21:02:57.827909 (+ 54us) client_negotiation.cc:240] Negotiation successful
0109 21:02:57.827967 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":79,"threads_started":1}
I20250109 21:02:57.828893 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:57.822556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:42921 (local address 127.0.0.1:43686)
0109 21:02:57.823392 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0109 21:02:57.823431 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0109 21:02:57.823603 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:02:57.824338 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:02:57.824345 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:02:57.824391 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:02:57.824715 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:57.824723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.826865 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:02:57.826868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:02:57.827711 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:57.827722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.828618 (+ 896us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:57.828634 (+ 16us) client_negotiation.cc:769] Sending connection context
0109 21:02:57.828718 (+ 84us) client_negotiation.cc:240] Negotiation successful
0109 21:02:57.828777 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"spinlock_wait_cycles":7040,"thread_start_us":93,"threads_started":1}
I20250109 21:02:57.829030 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:57.822701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43686 (local address 127.12.126.65:42921)
0109 21:02:57.823129 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:57.823136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:57.823525 (+ 389us) server_negotiation.cc:407] Connection header received
0109 21:02:57.823791 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:57.823797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:57.823976 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:57.824109 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:02:57.824881 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.826068 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:57.827885 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.828442 (+ 557us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:57.828550 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:57.828793 (+ 243us) server_negotiation.cc:299] Negotiation successful
0109 21:02:57.828894 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":145,"threads_started":1}
I20250109 21:02:57.829254 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:57.823556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51396 (local address 127.12.126.66:45965)
0109 21:02:57.824045 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:57.824050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:57.824071 (+ 21us) server_negotiation.cc:407] Connection header received
0109 21:02:57.824155 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:57.824160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:57.824301 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:57.824438 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:02:57.824946 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.826069 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:57.828337 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:57.828863 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:57.828973 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:57.829050 (+ 77us) server_negotiation.cc:299] Negotiation successful
0109 21:02:57.829148 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":93,"threads_started":1}
I20250109 21:02:57.830449 13756 tablet_service.cc:1467] Processing CreateTablet for tablet bde324092251458ba5a831a79ad3249f (DEFAULT_TABLE table=test-table [id=fd2a9932edb74449a8b1f24599315c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:02:57.831529 13756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bde324092251458ba5a831a79ad3249f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:02:57.831240 13488 tablet_service.cc:1467] Processing CreateTablet for tablet bde324092251458ba5a831a79ad3249f (DEFAULT_TABLE table=test-table [id=fd2a9932edb74449a8b1f24599315c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:02:57.831523 13622 tablet_service.cc:1467] Processing CreateTablet for tablet bde324092251458ba5a831a79ad3249f (DEFAULT_TABLE table=test-table [id=fd2a9932edb74449a8b1f24599315c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:02:57.832165 13488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bde324092251458ba5a831a79ad3249f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:02:57.832358 13622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bde324092251458ba5a831a79ad3249f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:02:57.837589 13839 tablet_bootstrap.cc:492] T bde324092251458ba5a831a79ad3249f P 84552845e3b34c45ad6e0018326ab883: Bootstrap starting.
I20250109 21:02:57.838199 13840 tablet_bootstrap.cc:492] T bde324092251458ba5a831a79ad3249f P 8365b1ef67ad41acaa7b4821938f644a: Bootstrap starting.
I20250109 21:02:57.839373 13841 tablet_bootstrap.cc:492] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49: Bootstrap starting.
I20250109 21:02:57.840299 13839 tablet_bootstrap.cc:654] T bde324092251458ba5a831a79ad3249f P 84552845e3b34c45ad6e0018326ab883: Neither blocks nor log segments found. Creating new log.
I20250109 21:02:57.840298 13840 tablet_bootstrap.cc:654] T bde324092251458ba5a831a79ad3249f P 8365b1ef67ad41acaa7b4821938f644a: Neither blocks nor log segments found. Creating new log.
I20250109 21:02:57.841250 13839 log.cc:826] T bde324092251458ba5a831a79ad3249f P 84552845e3b34c45ad6e0018326ab883: Log is configured to *not* fsync() on all Append() calls
I20250109 21:02:57.841305 13841 tablet_bootstrap.cc:654] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49: Neither blocks nor log segments found. Creating new log.
I20250109 21:02:57.841580 13840 log.cc:826] T bde324092251458ba5a831a79ad3249f P 8365b1ef67ad41acaa7b4821938f644a: Log is configured to *not* fsync() on all Append() calls
I20250109 21:02:57.842497 13841 log.cc:826] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49: Log is configured to *not* fsync() on all Append() calls
I20250109 21:02:57.843611 13839 tablet_bootstrap.cc:492] T bde324092251458ba5a831a79ad3249f P 84552845e3b34c45ad6e0018326ab883: No bootstrap required, opened a new log
I20250109 21:02:57.843641 13840 tablet_bootstrap.cc:492] T bde324092251458ba5a831a79ad3249f P 8365b1ef67ad41acaa7b4821938f644a: No bootstrap required, opened a new log
I20250109 21:02:57.843797 13839 ts_tablet_manager.cc:1397] T bde324092251458ba5a831a79ad3249f P 84552845e3b34c45ad6e0018326ab883: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250109 21:02:57.843809 13840 ts_tablet_manager.cc:1397] T bde324092251458ba5a831a79ad3249f P 8365b1ef67ad41acaa7b4821938f644a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250109 21:02:57.844188 13841 tablet_bootstrap.cc:492] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49: No bootstrap required, opened a new log
I20250109 21:02:57.844350 13841 ts_tablet_manager.cc:1397] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250109 21:02:57.846560 13839 raft_consensus.cc:357] T bde324092251458ba5a831a79ad3249f P 84552845e3b34c45ad6e0018326ab883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84552845e3b34c45ad6e0018326ab883" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42921 } } peers { permanent_uuid: "8365b1ef67ad41acaa7b4821938f644a" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 35893 } } peers { permanent_uuid: "a39f5deee7db432fab6cb589c0b2bf49" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45965 } }
I20250109 21:02:57.846818 13839 raft_consensus.cc:383] T bde324092251458ba5a831a79ad3249f P 84552845e3b34c45ad6e0018326ab883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:02:57.846879 13839 raft_consensus.cc:738] T bde324092251458ba5a831a79ad3249f P 84552845e3b34c45ad6e0018326ab883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84552845e3b34c45ad6e0018326ab883, State: Initialized, Role: FOLLOWER
I20250109 21:02:57.847308 13839 consensus_queue.cc:260] T bde324092251458ba5a831a79ad3249f P 84552845e3b34c45ad6e0018326ab883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84552845e3b34c45ad6e0018326ab883" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42921 } } peers { permanent_uuid: "8365b1ef67ad41acaa7b4821938f644a" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 35893 } } peers { permanent_uuid: "a39f5deee7db432fab6cb589c0b2bf49" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45965 } }
I20250109 21:02:57.847290 13840 raft_consensus.cc:357] T bde324092251458ba5a831a79ad3249f P 8365b1ef67ad41acaa7b4821938f644a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84552845e3b34c45ad6e0018326ab883" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42921 } } peers { permanent_uuid: "8365b1ef67ad41acaa7b4821938f644a" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 35893 } } peers { permanent_uuid: "a39f5deee7db432fab6cb589c0b2bf49" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45965 } }
I20250109 21:02:57.847487 13840 raft_consensus.cc:383] T bde324092251458ba5a831a79ad3249f P 8365b1ef67ad41acaa7b4821938f644a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:02:57.847545 13840 raft_consensus.cc:738] T bde324092251458ba5a831a79ad3249f P 8365b1ef67ad41acaa7b4821938f644a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8365b1ef67ad41acaa7b4821938f644a, State: Initialized, Role: FOLLOWER
I20250109 21:02:57.847541 13841 raft_consensus.cc:357] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84552845e3b34c45ad6e0018326ab883" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42921 } } peers { permanent_uuid: "8365b1ef67ad41acaa7b4821938f644a" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 35893 } } peers { permanent_uuid: "a39f5deee7db432fab6cb589c0b2bf49" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45965 } }
I20250109 21:02:57.847721 13841 raft_consensus.cc:383] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:02:57.847784 13841 raft_consensus.cc:738] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39f5deee7db432fab6cb589c0b2bf49, State: Initialized, Role: FOLLOWER
I20250109 21:02:57.847993 13840 consensus_queue.cc:260] T bde324092251458ba5a831a79ad3249f P 8365b1ef67ad41acaa7b4821938f644a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84552845e3b34c45ad6e0018326ab883" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42921 } } peers { permanent_uuid: "8365b1ef67ad41acaa7b4821938f644a" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 35893 } } peers { permanent_uuid: "a39f5deee7db432fab6cb589c0b2bf49" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45965 } }
I20250109 21:02:57.848119 13554 heartbeater.cc:502] Master 127.12.126.126:42889 was elected leader, sending a full tablet report...
I20250109 21:02:57.848213 13839 ts_tablet_manager.cc:1428] T bde324092251458ba5a831a79ad3249f P 84552845e3b34c45ad6e0018326ab883: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250109 21:02:57.848160 13841 consensus_queue.cc:260] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84552845e3b34c45ad6e0018326ab883" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42921 } } peers { permanent_uuid: "8365b1ef67ad41acaa7b4821938f644a" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 35893 } } peers { permanent_uuid: "a39f5deee7db432fab6cb589c0b2bf49" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45965 } }
I20250109 21:02:57.848843 13840 ts_tablet_manager.cc:1428] T bde324092251458ba5a831a79ad3249f P 8365b1ef67ad41acaa7b4821938f644a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:02:57.848891 13688 heartbeater.cc:502] Master 127.12.126.126:42889 was elected leader, sending a full tablet report...
I20250109 21:02:57.849232 13822 heartbeater.cc:502] Master 127.12.126.126:42889 was elected leader, sending a full tablet report...
I20250109 21:02:57.849802 13841 ts_tablet_manager.cc:1428] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
W20250109 21:02:58.032413 13823 tablet.cc:2367] T bde324092251458ba5a831a79ad3249f P 8365b1ef67ad41acaa7b4821938f644a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:02:58.059599 13847 raft_consensus.cc:491] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:02:58.059823 13847 raft_consensus.cc:513] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84552845e3b34c45ad6e0018326ab883" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42921 } } peers { permanent_uuid: "8365b1ef67ad41acaa7b4821938f644a" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 35893 } } peers { permanent_uuid: "a39f5deee7db432fab6cb589c0b2bf49" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45965 } }
I20250109 21:02:58.061149 13847 leader_election.cc:290] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84552845e3b34c45ad6e0018326ab883 (127.12.126.65:42921), 8365b1ef67ad41acaa7b4821938f644a (127.12.126.67:35893)
I20250109 21:02:58.064465 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:58.061404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:35893 (local address 127.12.126.66:33053)
0109 21:02:58.061575 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0109 21:02:58.061592 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0109 21:02:58.061688 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:02:58.062254 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:02:58.062257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:02:58.062273 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:02:58.062490 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:58.062497 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:58.063403 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:02:58.063408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:02:58.064046 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:58.064055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:58.064165 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:58.064189 (+ 24us) client_negotiation.cc:769] Sending connection context
0109 21:02:58.064243 (+ 54us) client_negotiation.cc:240] Negotiation successful
0109 21:02:58.064301 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
W20250109 21:02:58.065066 13689 tablet.cc:2367] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:02:58.065243 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:58.061404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:42921 (local address 127.12.126.66:60371)
0109 21:02:58.061849 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0109 21:02:58.061861 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:02:58.061958 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:02:58.062606 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:02:58.062609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:02:58.062627 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:02:58.062928 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:58.062935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:58.063929 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:02:58.063932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:02:58.064877 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:58.064885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:58.064975 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:58.064991 (+ 16us) client_negotiation.cc:769] Sending connection context
0109 21:02:58.065037 (+ 46us) client_negotiation.cc:240] Negotiation successful
0109 21:02:58.065102 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":121,"thread_start_us":87,"threads_started":1}
I20250109 21:02:58.065253 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:58.061842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:33053 (local address 127.12.126.67:35893)
0109 21:02:58.062009 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:58.062015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:58.062030 (+ 15us) server_negotiation.cc:407] Connection header received
0109 21:02:58.062086 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:58.062088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:58.062128 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:58.062231 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:02:58.062608 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:58.063296 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:58.064175 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:58.064924 (+ 749us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:58.064963 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:58.065015 (+ 52us) server_negotiation.cc:299] Negotiation successful
0109 21:02:58.065068 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20250109 21:02:58.066248 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:58.061635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:60371 (local address 127.12.126.65:42921)
0109 21:02:58.061776 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:58.061781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:58.061938 (+ 157us) server_negotiation.cc:407] Connection header received
0109 21:02:58.062079 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:58.062082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:58.062134 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:58.062232 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:02:58.063038 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:58.063804 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:58.065390 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:58.065952 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:58.065996 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:58.066052 (+ 56us) server_negotiation.cc:299] Negotiation successful
0109 21:02:58.066112 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250109 21:02:58.066268 13776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde324092251458ba5a831a79ad3249f" candidate_uuid: "a39f5deee7db432fab6cb589c0b2bf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8365b1ef67ad41acaa7b4821938f644a" is_pre_election: true
I20250109 21:02:58.066596 13776 raft_consensus.cc:2463] T bde324092251458ba5a831a79ad3249f P 8365b1ef67ad41acaa7b4821938f644a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a39f5deee7db432fab6cb589c0b2bf49 in term 0.
I20250109 21:02:58.066830 13508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde324092251458ba5a831a79ad3249f" candidate_uuid: "a39f5deee7db432fab6cb589c0b2bf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84552845e3b34c45ad6e0018326ab883" is_pre_election: true
I20250109 21:02:58.067087 13508 raft_consensus.cc:2463] T bde324092251458ba5a831a79ad3249f P 84552845e3b34c45ad6e0018326ab883 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a39f5deee7db432fab6cb589c0b2bf49 in term 0.
I20250109 21:02:58.067112 13578 leader_election.cc:304] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8365b1ef67ad41acaa7b4821938f644a, a39f5deee7db432fab6cb589c0b2bf49; no voters:
I20250109 21:02:58.067348 13847 raft_consensus.cc:2798] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:02:58.067447 13847 raft_consensus.cc:491] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:02:58.067526 13847 raft_consensus.cc:3054] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:02:58.068506 13847 raft_consensus.cc:513] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84552845e3b34c45ad6e0018326ab883" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42921 } } peers { permanent_uuid: "8365b1ef67ad41acaa7b4821938f644a" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 35893 } } peers { permanent_uuid: "a39f5deee7db432fab6cb589c0b2bf49" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45965 } }
I20250109 21:02:58.068874 13847 leader_election.cc:290] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49 [CANDIDATE]: Term 1 election: Requested vote from peers 84552845e3b34c45ad6e0018326ab883 (127.12.126.65:42921), 8365b1ef67ad41acaa7b4821938f644a (127.12.126.67:35893)
I20250109 21:02:58.069244 13508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde324092251458ba5a831a79ad3249f" candidate_uuid: "a39f5deee7db432fab6cb589c0b2bf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84552845e3b34c45ad6e0018326ab883"
I20250109 21:02:58.069231 13776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde324092251458ba5a831a79ad3249f" candidate_uuid: "a39f5deee7db432fab6cb589c0b2bf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8365b1ef67ad41acaa7b4821938f644a"
I20250109 21:02:58.069358 13508 raft_consensus.cc:3054] T bde324092251458ba5a831a79ad3249f P 84552845e3b34c45ad6e0018326ab883 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:02:58.069360 13776 raft_consensus.cc:3054] T bde324092251458ba5a831a79ad3249f P 8365b1ef67ad41acaa7b4821938f644a [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:02:58.070596 13776 raft_consensus.cc:2463] T bde324092251458ba5a831a79ad3249f P 8365b1ef67ad41acaa7b4821938f644a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39f5deee7db432fab6cb589c0b2bf49 in term 1.
I20250109 21:02:58.070596 13508 raft_consensus.cc:2463] T bde324092251458ba5a831a79ad3249f P 84552845e3b34c45ad6e0018326ab883 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39f5deee7db432fab6cb589c0b2bf49 in term 1.
I20250109 21:02:58.070922 13578 leader_election.cc:304] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8365b1ef67ad41acaa7b4821938f644a, a39f5deee7db432fab6cb589c0b2bf49; no voters:
I20250109 21:02:58.071128 13847 raft_consensus.cc:2798] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:02:58.071337 13847 raft_consensus.cc:695] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49 [term 1 LEADER]: Becoming Leader. State: Replica: a39f5deee7db432fab6cb589c0b2bf49, State: Running, Role: LEADER
I20250109 21:02:58.071659 13847 consensus_queue.cc:237] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84552845e3b34c45ad6e0018326ab883" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42921 } } peers { permanent_uuid: "8365b1ef67ad41acaa7b4821938f644a" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 35893 } } peers { permanent_uuid: "a39f5deee7db432fab6cb589c0b2bf49" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45965 } }
I20250109 21:02:58.074690 13362 catalog_manager.cc:5526] T bde324092251458ba5a831a79ad3249f P a39f5deee7db432fab6cb589c0b2bf49 reported cstate change: term changed from 0 to 1, leader changed from <none> to a39f5deee7db432fab6cb589c0b2bf49 (127.12.126.66). New cstate: current_term: 1 leader_uuid: "a39f5deee7db432fab6cb589c0b2bf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84552845e3b34c45ad6e0018326ab883" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8365b1ef67ad41acaa7b4821938f644a" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 35893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a39f5deee7db432fab6cb589c0b2bf49" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45965 } health_report { overall_health: HEALTHY } } }
W20250109 21:02:58.088955 13555 tablet.cc:2367] T bde324092251458ba5a831a79ad3249f P 84552845e3b34c45ad6e0018326ab883: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jan 09 21:02:58 dist-test-slave-g0zb krb5kdc[13318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.65@KRBTEST.COM
I20250109 21:02:58.123888 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:58.115992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43702 (local address 127.12.126.65:42921)
0109 21:02:58.116152 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:58.116156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:58.116221 (+ 65us) server_negotiation.cc:407] Connection header received
0109 21:02:58.116343 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:58.116346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:58.116390 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:58.116462 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:02:58.117219 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:58.117712 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:58.118307 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:58.118444 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:58.120661 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:02:58.120686 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:02:58.120700 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:02:58.120724 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:02:58.122488 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:58.122916 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:58.122921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:58.122923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:58.122972 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:58.123142 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:58.123144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:58.123145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:58.123332 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:02:58.123494 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:58.123596 (+ 102us) server_negotiation.cc:299] Negotiation successful
0109 21:02:58.123694 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Jan 09 21:02:58 dist-test-slave-g0zb krb5kdc[13318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456578, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 09 21:02:58 dist-test-slave-g0zb krb5kdc[13318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456578, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.126.65@KRBTEST.COM
I20250109 21:02:58.147917 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:58.140355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.12.126.65:42921)
0109 21:02:58.140495 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:58.140500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:58.140595 (+ 95us) server_negotiation.cc:407] Connection header received
0109 21:02:58.140762 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:58.140765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:58.140813 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:58.140886 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:02:58.141695 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:58.142169 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:58.142993 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:58.143205 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:58.145218 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:02:58.145250 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:02:58.145253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:02:58.145280 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:02:58.146509 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:58.146886 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:58.146888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:58.146890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:58.146931 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:58.147182 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:58.147184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:58.147185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:58.147323 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:02:58.147438 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:58.147643 (+ 205us) server_negotiation.cc:299] Negotiation successful
0109 21:02:58.147735 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Jan 09 21:02:58 dist-test-slave-g0zb krb5kdc[13318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 09 21:02:58 dist-test-slave-g0zb krb5kdc[13318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.65@KRBTEST.COM
I20250109 21:02:58.170779 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:58.163390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43722 (local address 127.12.126.65:42921)
0109 21:02:58.163533 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:58.163538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:58.163625 (+ 87us) server_negotiation.cc:407] Connection header received
0109 21:02:58.163784 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:58.163787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:58.163832 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:58.163905 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:02:58.164693 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:58.165147 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:58.165784 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:58.165990 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:58.168124 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:02:58.168142 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:02:58.168145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:02:58.168172 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:02:58.169546 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:58.169924 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:58.169926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:58.169928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:58.169964 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:58.170118 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:58.170120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:58.170121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:58.170252 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:02:58.170346 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:58.170564 (+ 218us) server_negotiation.cc:299] Negotiation successful
0109 21:02:58.170654 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20250109 21:02:58.172134 13468 tablet_service.cc:2908] Rejecting scan request for tablet bde324092251458ba5a831a79ad3249f: Uninitialized: clean time has not yet been initialized
I20250109 21:02:58.173822 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 13425
I20250109 21:02:58.179904 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 13559
I20250109 21:02:58.185696 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 13693
I20250109 21:02:58.191468 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 13331
2025-01-09T21:02:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2458 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:02:58 dist-test-slave-g0zb krb5kdc[13863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:02:58 dist-test-slave-g0zb krb5kdc[13863](info): set up 2 sockets
Jan 09 21:02:58 dist-test-slave-g0zb krb5kdc[13863](info): commencing operation
krb5kdc: starting...
W20250109 21:02:59.223222 12793 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:02:59 dist-test-slave-g0zb krb5kdc[13863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:02:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:02:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:02:59.341403 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:42839
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:36629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:42839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:02:59.447952 13876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:02:59.448205 13876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:02:59.448295 13876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:02:59.451409 13876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:02:59.451467 13876 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250109 21:02:59.451498 13876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:02:59.451534 13876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:02:59.451575 13876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:02:59.451615 13876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:02:59.455931 13876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:36629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:42839
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:42839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:02:59.457000 13876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:02:59.457845 13876 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:02:59.464516 13882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:59.465322 13884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:59.466001 13881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:02:59.466677 13876 server_base.cc:1034] running on GCE node
I20250109 21:02:59.467211 13876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:02:59.468169 13876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:02:59.469374 13876 hybrid_clock.cc:648] HybridClock initialized: now 1736456579469355 us; error 37 us; skew 500 ppm
Jan 09 21:02:59 dist-test-slave-g0zb krb5kdc[13863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456579, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:02:59.477227 13876 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:02:59.478122 13876 webserver.cc:458] Webserver started at http://127.12.126.126:42049/ using document root <none> and password file <none>
I20250109 21:02:59.478653 13876 fs_manager.cc:362] Metadata directory not provided
I20250109 21:02:59.478704 13876 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:02:59.478904 13876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:02:59.480674 13876 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "7e0fd249788a4a8ea6183baf6fb3dfb8"
format_stamp: "Formatted at 2025-01-09 21:02:59 on dist-test-slave-g0zb"
server_key: "b2005321e8895c6a29cde93b929e60e8"
server_key_iv: "89169026d0951ce77be0f70232a53483"
server_key_version: "encryptionkey@0"
I20250109 21:02:59.481105 13876 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "7e0fd249788a4a8ea6183baf6fb3dfb8"
format_stamp: "Formatted at 2025-01-09 21:02:59 on dist-test-slave-g0zb"
server_key: "b2005321e8895c6a29cde93b929e60e8"
server_key_iv: "89169026d0951ce77be0f70232a53483"
server_key_version: "encryptionkey@0"
I20250109 21:02:59.484463 13876 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20250109 21:02:59.486584 13891 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:59.487620 13876 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:02:59.487732 13876 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "7e0fd249788a4a8ea6183baf6fb3dfb8"
format_stamp: "Formatted at 2025-01-09 21:02:59 on dist-test-slave-g0zb"
server_key: "b2005321e8895c6a29cde93b929e60e8"
server_key_iv: "89169026d0951ce77be0f70232a53483"
server_key_version: "encryptionkey@0"
I20250109 21:02:59.487818 13876 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:02:59.519759 13876 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:02:59.522591 13876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:02:59.522768 13876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:02:59.530457 13876 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:42839
I20250109 21:02:59.530472 13953 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:42839 every 8 connection(s)
I20250109 21:02:59.531500 13876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:02:59.534070 13954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:02:59.537052 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 13876
I20250109 21:02:59.537158 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:02:59.537420 12793 external_mini_cluster.cc:1453] Setting key 982a790bc2a3764003e7c311b8b44ac2
I20250109 21:02:59.539474 13954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8: Bootstrap starting.
I20250109 21:02:59.541544 13954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8: Neither blocks nor log segments found. Creating new log.
I20250109 21:02:59.542253 13954 log.cc:826] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8: Log is configured to *not* fsync() on all Append() calls
I20250109 21:02:59.544128 13954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8: No bootstrap required, opened a new log
Jan 09 21:02:59 dist-test-slave-g0zb krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:02:59.547518 13954 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0fd249788a4a8ea6183baf6fb3dfb8" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42839 } }
I20250109 21:02:59.547708 13954 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:02:59.547782 13954 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e0fd249788a4a8ea6183baf6fb3dfb8, State: Initialized, Role: FOLLOWER
I20250109 21:02:59.548275 13954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0fd249788a4a8ea6183baf6fb3dfb8" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42839 } }
I20250109 21:02:59.548441 13954 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:02:59.548507 13954 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:02:59.548626 13954 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:02:59.549703 13954 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0fd249788a4a8ea6183baf6fb3dfb8" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42839 } }
I20250109 21:02:59.550058 13954 leader_election.cc:304] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e0fd249788a4a8ea6183baf6fb3dfb8; no voters:
I20250109 21:02:59.550335 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:59.538703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51626 (local address 127.12.126.126:42839)
0109 21:02:59.539219 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:59.539229 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:59.539264 (+ 35us) server_negotiation.cc:407] Connection header received
0109 21:02:59.539869 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:59.539888 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:59.540191 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:59.540537 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:02:59.541411 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:59.542295 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:59.542978 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:59.543289 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:59.545391 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:02:59.545414 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:02:59.545429 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:02:59.545486 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:02:59.547466 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:59.547913 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:59.547919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:59.547924 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:59.548020 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:59.548253 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:59.548256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:59.548258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:59.548733 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:02:59.548963 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:59.549258 (+ 295us) server_negotiation.cc:299] Negotiation successful
0109 21:02:59.549541 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":135,"threads_started":1}
I20250109 21:02:59.550364 13954 leader_election.cc:290] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:02:59.550554 13959 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:02:59.550760 13959 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8 [term 1 LEADER]: Becoming Leader. State: Replica: 7e0fd249788a4a8ea6183baf6fb3dfb8, State: Running, Role: LEADER
I20250109 21:02:59.551098 13959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0fd249788a4a8ea6183baf6fb3dfb8" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42839 } }
I20250109 21:02:59.551529 13954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:02:59.552234 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e0fd249788a4a8ea6183baf6fb3dfb8. Latest consensus state: current_term: 1 leader_uuid: "7e0fd249788a4a8ea6183baf6fb3dfb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0fd249788a4a8ea6183baf6fb3dfb8" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42839 } } }
I20250109 21:02:59.552345 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8 [sys.catalog]: This master's current role is: LEADER
I20250109 21:02:59.553241 13960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e0fd249788a4a8ea6183baf6fb3dfb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0fd249788a4a8ea6183baf6fb3dfb8" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42839 } } }
I20250109 21:02:59.553340 13960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8 [sys.catalog]: This master's current role is: LEADER
W20250109 21:02:59.556138 13974 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:02:59.556216 13974 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 21:02:59.556293 13968 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:02:59.557106 13968 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:02:59.562623 13968 catalog_manager.cc:1348] Generated new cluster ID: ae740678418d41a9854cb6e1ce9e7a1d
I20250109 21:02:59.562691 13968 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:02:59.582491 13968 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:02:59.583505 13968 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:02:59.591246 13968 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8: Generated new TSK 0
I20250109 21:02:59.591996 13968 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
I20250109 21:02:59.622190 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42839
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:36629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:02:59.728163 13980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:02:59.728390 13980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:02:59.728451 13980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:02:59.731493 13980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:02:59.731542 13980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:02:59.731590 13980 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250109 21:02:59.731637 13980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:02:59.735677 13980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:36629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.126.126:42839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:02:59.736662 13980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:02:59.737444 13980 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:02:59.744928 13986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:59.745816 13985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:59.746428 13988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:02:59.747346 13980 server_base.cc:1034] running on GCE node
I20250109 21:02:59.747709 13980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:02:59.748226 13980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:02:59.749368 13980 hybrid_clock.cc:648] HybridClock initialized: now 1736456579749357 us; error 28 us; skew 500 ppm
Jan 09 21:02:59 dist-test-slave-g0zb krb5kdc[13863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456579, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:02:59.751997 13980 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:02:59.752899 13980 webserver.cc:458] Webserver started at http://127.12.126.65:34037/ using document root <none> and password file <none>
I20250109 21:02:59.753479 13980 fs_manager.cc:362] Metadata directory not provided
I20250109 21:02:59.753535 13980 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:02:59.753693 13980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:02:59.755486 13980 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "f9c542d489c14eaabf0bf9b425dbaf53"
format_stamp: "Formatted at 2025-01-09 21:02:59 on dist-test-slave-g0zb"
server_key: "96e86dd8d97d859767018824e04bcc4d"
server_key_iv: "44b046afbcc5944d78a1017f1f24cdac"
server_key_version: "encryptionkey@0"
I20250109 21:02:59.755910 13980 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "f9c542d489c14eaabf0bf9b425dbaf53"
format_stamp: "Formatted at 2025-01-09 21:02:59 on dist-test-slave-g0zb"
server_key: "96e86dd8d97d859767018824e04bcc4d"
server_key_iv: "44b046afbcc5944d78a1017f1f24cdac"
server_key_version: "encryptionkey@0"
I20250109 21:02:59.759224 13980 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:02:59.761261 13995 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:59.762228 13980 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:02:59.762315 13980 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "f9c542d489c14eaabf0bf9b425dbaf53"
format_stamp: "Formatted at 2025-01-09 21:02:59 on dist-test-slave-g0zb"
server_key: "96e86dd8d97d859767018824e04bcc4d"
server_key_iv: "44b046afbcc5944d78a1017f1f24cdac"
server_key_version: "encryptionkey@0"
I20250109 21:02:59.762390 13980 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:02:59.792822 13980 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:02:59.795502 13980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:02:59.795644 13980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:02:59.796934 13980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:02:59.796985 13980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:59.797029 13980 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:02:59.797075 13980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 09 21:02:59 dist-test-slave-g0zb krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456579, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:02:59.807402 13980 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:39535
I20250109 21:02:59.807566 14110 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:39535 every 8 connection(s)
I20250109 21:02:59.808363 13980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:02:59.809587 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:59.797535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:34567 (local address 127.12.126.126:42839)
0109 21:02:59.797659 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:59.797663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:59.798511 (+ 848us) server_negotiation.cc:407] Connection header received
0109 21:02:59.799391 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:59.799395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:59.799459 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:59.799545 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:02:59.800924 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:59.801846 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:59.802518 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:59.802703 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:59.806074 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:02:59.806096 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:02:59.806098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:02:59.806125 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:02:59.807696 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:59.808178 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:59.808182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:59.808183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:59.808228 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:59.808515 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:59.808517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:59.808519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:59.808662 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:02:59.808752 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:59.809312 (+ 560us) server_negotiation.cc:299] Negotiation successful
0109 21:02:59.809433 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250109 21:02:59.810160 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:59.797932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42839 (local address 127.12.126.65:34567)
0109 21:02:59.798381 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0109 21:02:59.798414 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:02:59.799144 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:02:59.799818 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:02:59.799826 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:02:59.800188 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:02:59.800766 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:59.800782 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:59.801978 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:02:59.801982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:02:59.802399 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:59.802408 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:59.802632 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:59.803274 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:02:59.803314 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:02:59.805911 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:02:59.807808 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:02:59.807816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:02:59.807830 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:02:59.808076 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:02:59.808314 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:02:59.808320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:02:59.808322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:02:59.808407 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:02:59.808768 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:02:59.808776 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:02:59.809013 (+ 237us) client_negotiation.cc:769] Sending connection context
0109 21:02:59.809250 (+ 237us) client_negotiation.cc:240] Negotiation successful
0109 21:02:59.809497 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":131,"threads_started":1}
I20250109 21:02:59.811357 14111 heartbeater.cc:346] Connected to a master server at 127.12.126.126:42839
I20250109 21:02:59.811477 14111 heartbeater.cc:463] Registering TS with master...
I20250109 21:02:59.811935 14111 heartbeater.cc:510] Master 127.12.126.126:42839 requested a full tablet report, sending...
I20250109 21:02:59.813442 13907 ts_manager.cc:194] Registered new tserver with Master: f9c542d489c14eaabf0bf9b425dbaf53 (127.12.126.65:39535)
I20250109 21:02:59.814594 13907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:34567
I20250109 21:02:59.817783 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 13980
I20250109 21:02:59.817889 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:02:59.818169 12793 external_mini_cluster.cc:1453] Setting key bcc247f2f357afbd4d2ba20eca61e667
I20250109 21:02:59.819797 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:59.813390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:48267 (local address 127.12.126.126:42839)
0109 21:02:59.813504 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0109 21:02:59.813511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:02:59.813689 (+ 178us) server_negotiation.cc:407] Connection header received
0109 21:02:59.813823 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:02:59.813825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:02:59.813865 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:02:59.813928 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:02:59.814438 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:59.814964 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:02:59.815677 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:02:59.815807 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:59.816658 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:02:59.816672 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:02:59.816674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:02:59.816694 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:02:59.818220 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:59.818761 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:59.818765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:59.818767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:59.818842 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:02:59.819073 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:02:59.819075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:02:59.819076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:02:59.819198 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:02:59.819271 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0109 21:02:59.819545 (+ 274us) server_negotiation.cc:299] Negotiation successful
0109 21:02:59.819652 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20250109 21:02:59.819806 14117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:02:59.813345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42839 (local address 127.12.126.65:48267)
0109 21:02:59.813612 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0109 21:02:59.813624 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:02:59.813718 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:02:59.813953 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:02:59.813956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:02:59.814117 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:02:59.814328 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:59.814335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:59.815081 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:02:59.815085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:02:59.815567 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:02:59.815578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:02:59.815691 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:02:59.816176 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:02:59.816191 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:02:59.816543 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:02:59.818338 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:02:59.818343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:02:59.818346 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:02:59.818646 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:02:59.818935 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:02:59.818938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:02:59.818940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:02:59.818997 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:02:59.819277 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:02:59.819280 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:02:59.819360 (+ 80us) client_negotiation.cc:769] Sending connection context
0109 21:02:59.819481 (+ 121us) client_negotiation.cc:240] Negotiation successful
0109 21:02:59.819617 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.12.126.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
I20250109 21:02:59.843895 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42839
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:36629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:02:59.949427 14120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:02:59.949645 14120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:02:59.949704 14120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:02:59.952775 14120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:02:59.952831 14120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:02:59.952883 14120 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250109 21:02:59.952930 14120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:02:59.957178 14120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:36629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.126.126:42839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:02:59.958192 14120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:02:59.959064 14120 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:02:59.967542 14126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:59.968261 14128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:02:59.968358 14125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:02:59.969004 14120 server_base.cc:1034] running on GCE node
I20250109 21:02:59.969337 14120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:02:59.969880 14120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:02:59.971025 14120 hybrid_clock.cc:648] HybridClock initialized: now 1736456579971012 us; error 27 us; skew 500 ppm
Jan 09 21:02:59 dist-test-slave-g0zb krb5kdc[13863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456579, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:02:59.973618 14120 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:02:59.974467 14120 webserver.cc:458] Webserver started at http://127.12.126.66:40713/ using document root <none> and password file <none>
I20250109 21:02:59.975042 14120 fs_manager.cc:362] Metadata directory not provided
I20250109 21:02:59.975096 14120 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:02:59.975281 14120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:02:59.977030 14120 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "e5fd5416a3094aefbbb4d2cc835c3736"
format_stamp: "Formatted at 2025-01-09 21:02:59 on dist-test-slave-g0zb"
server_key: "8343d285883e8204a580f281464d084d"
server_key_iv: "cf7bec5a6cae3e25822dc221d514c9b0"
server_key_version: "encryptionkey@0"
I20250109 21:02:59.977505 14120 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "e5fd5416a3094aefbbb4d2cc835c3736"
format_stamp: "Formatted at 2025-01-09 21:02:59 on dist-test-slave-g0zb"
server_key: "8343d285883e8204a580f281464d084d"
server_key_iv: "cf7bec5a6cae3e25822dc221d514c9b0"
server_key_version: "encryptionkey@0"
I20250109 21:02:59.980870 14120 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:02:59.982964 14135 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:02:59.983952 14120 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:02:59.984056 14120 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "e5fd5416a3094aefbbb4d2cc835c3736"
format_stamp: "Formatted at 2025-01-09 21:02:59 on dist-test-slave-g0zb"
server_key: "8343d285883e8204a580f281464d084d"
server_key_iv: "cf7bec5a6cae3e25822dc221d514c9b0"
server_key_version: "encryptionkey@0"
I20250109 21:02:59.984148 14120 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:02:59.997129 14120 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:00.000172 14120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:00.000335 14120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:00.001659 14120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:00.001713 14120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:00.001777 14120 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:00.001808 14120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 09 21:03:00 dist-test-slave-g0zb krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456579, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:00.011934 14120 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:43019
I20250109 21:03:00.011956 14250 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:43019 every 8 connection(s)
I20250109 21:03:00.013101 14120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:03:00.015182 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.002287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:38069 (local address 127.12.126.126:42839)
0109 21:03:00.002407 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.002411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.003260 (+ 849us) server_negotiation.cc:407] Connection header received
0109 21:03:00.004052 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.004056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.004118 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.004208 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:00.005978 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.006794 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.007768 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.007973 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.011232 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:00.011253 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:00.011256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:00.011285 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:00.013134 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:00.013686 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:00.013692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:00.013694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:00.013753 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:00.014148 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:00.014151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:00.014153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:00.014335 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:00.014437 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.014948 (+ 511us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.015059 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250109 21:03:00.015820 14144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.002661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42839 (local address 127.12.126.66:38069)
0109 21:03:00.003127 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:00.003161 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:00.003852 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:00.004425 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:00.004438 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:00.004975 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:00.005771 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.005792 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.006968 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.006973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:00.007641 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.007654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.007894 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.008578 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:00.008610 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:00.011067 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:00.013284 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:00.013288 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:00.013297 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:00.013571 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:00.013859 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:00.013862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:00.013864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:00.013950 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:00.014438 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:00.014444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:00.014694 (+ 250us) client_negotiation.cc:769] Sending connection context
0109 21:03:00.014897 (+ 203us) client_negotiation.cc:240] Negotiation successful
0109 21:03:00.015141 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":147,"threads_started":1}
I20250109 21:03:00.017011 14251 heartbeater.cc:346] Connected to a master server at 127.12.126.126:42839
I20250109 21:03:00.017148 14251 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:00.017678 14251 heartbeater.cc:510] Master 127.12.126.126:42839 requested a full tablet report, sending...
I20250109 21:03:00.018739 13908 ts_manager.cc:194] Registered new tserver with Master: e5fd5416a3094aefbbb4d2cc835c3736 (127.12.126.66:43019)
I20250109 21:03:00.019405 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 14120
I20250109 21:03:00.019539 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:03:00.019516 13908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:38069
I20250109 21:03:00.019856 12793 external_mini_cluster.cc:1453] Setting key a969f8afa214a82e8faad8ab6c672267
I20250109 21:03:00.025581 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.018995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:59259 (local address 127.12.126.126:42839)
0109 21:03:00.019148 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.019152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.019374 (+ 222us) server_negotiation.cc:407] Connection header received
0109 21:03:00.019483 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.019488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.019543 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.019634 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:00.020263 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.020903 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.021568 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.021788 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.022523 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:00.022544 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:00.022551 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:00.022582 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:00.024053 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:00.024535 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:00.024540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:00.024543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:00.024595 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:00.024831 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:00.024834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:00.024836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:00.024993 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:00.025085 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.025324 (+ 239us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.025430 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250109 21:03:00.025604 14257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.019014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42839 (local address 127.12.126.66:59259)
0109 21:03:00.019275 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:00.019290 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:00.019388 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:00.019661 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:00.019664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:00.019861 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:00.020124 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.020132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.021043 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.021046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:00.021459 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.021476 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.021565 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.022099 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:00.022112 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:00.022398 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:00.024167 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:00.024173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:00.024177 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:00.024434 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:00.024695 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:00.024697 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:00.024699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:00.024748 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:00.025086 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:00.025091 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:00.025170 (+ 79us) client_negotiation.cc:769] Sending connection context
0109 21:03:00.025265 (+ 95us) client_negotiation.cc:240] Negotiation successful
0109 21:03:00.025411 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.12.126.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
I20250109 21:03:00.046861 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42839
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:36629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:00.152670 14260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:00.152943 14260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:00.153005 14260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:00.156201 14260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:00.156255 14260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:00.156306 14260 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250109 21:03:00.156353 14260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:03:00.160534 14260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:36629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.126.126:42839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:00.161520 14260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:00.162333 14260 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:00.170190 14265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:00.170921 14266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:00.171762 14268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:00.172386 14260 server_base.cc:1034] running on GCE node
I20250109 21:03:00.172744 14260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:00.173259 14260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:00.174401 14260 hybrid_clock.cc:648] HybridClock initialized: now 1736456580174380 us; error 40 us; skew 500 ppm
Jan 09 21:03:00 dist-test-slave-g0zb krb5kdc[13863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456580, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:00.177042 14260 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:03:00.177946 14260 webserver.cc:458] Webserver started at http://127.12.126.67:37399/ using document root <none> and password file <none>
I20250109 21:03:00.178493 14260 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:00.178553 14260 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:00.178723 14260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:00.180512 14260 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "3a09d6e0acc04084a3471c19e3b2d8d4"
format_stamp: "Formatted at 2025-01-09 21:03:00 on dist-test-slave-g0zb"
server_key: "f82352c95a210f38ad74d510ff11c220"
server_key_iv: "a7f0a5f5a1f88d9571147532d10902d2"
server_key_version: "encryptionkey@0"
I20250109 21:03:00.180984 14260 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "3a09d6e0acc04084a3471c19e3b2d8d4"
format_stamp: "Formatted at 2025-01-09 21:03:00 on dist-test-slave-g0zb"
server_key: "f82352c95a210f38ad74d510ff11c220"
server_key_iv: "a7f0a5f5a1f88d9571147532d10902d2"
server_key_version: "encryptionkey@0"
I20250109 21:03:00.184329 14260 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:03:00.186522 14275 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:00.187563 14260 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:00.187670 14260 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "3a09d6e0acc04084a3471c19e3b2d8d4"
format_stamp: "Formatted at 2025-01-09 21:03:00 on dist-test-slave-g0zb"
server_key: "f82352c95a210f38ad74d510ff11c220"
server_key_iv: "a7f0a5f5a1f88d9571147532d10902d2"
server_key_version: "encryptionkey@0"
I20250109 21:03:00.187759 14260 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:00.204638 14260 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:00.207392 14260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:00.207559 14260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:00.208866 14260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:00.208918 14260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:00.208978 14260 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:00.209007 14260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 09 21:03:00 dist-test-slave-g0zb krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456580, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:00.219489 14260 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:43645
I20250109 21:03:00.219498 14390 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:43645 every 8 connection(s)
I20250109 21:03:00.220629 14260 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:03:00.221967 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.209549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:47985 (local address 127.12.126.126:42839)
0109 21:03:00.209673 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.209677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.210502 (+ 825us) server_negotiation.cc:407] Connection header received
0109 21:03:00.211382 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.211386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.211464 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.211543 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:00.213403 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.213940 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.214696 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.214926 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.218236 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:00.218261 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:00.218266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:00.218297 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:00.219926 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:00.220406 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:00.220410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:00.220412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:00.220467 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:00.220744 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:00.220747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:00.220749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:00.220946 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:00.221063 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.221758 (+ 695us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.221854 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250109 21:03:00.222419 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 14260
I20250109 21:03:00.222434 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.209893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42839 (local address 127.12.126.67:47985)
0109 21:03:00.210360 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:00.210394 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:00.211139 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:00.211884 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:00.211896 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:00.212419 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:00.213097 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.213115 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.214113 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.214118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:00.214577 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.214586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.214825 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.215469 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:00.215504 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:00.218073 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:00.220055 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:00.220066 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:00.220074 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:00.220302 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:00.220557 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:00.220559 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:00.220560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:00.220629 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:00.221056 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:00.221062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:00.221361 (+ 299us) client_negotiation.cc:769] Sending connection context
0109 21:03:00.221569 (+ 208us) client_negotiation.cc:240] Negotiation successful
0109 21:03:00.221774 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":125,"threads_started":1}
I20250109 21:03:00.222517 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:03:00.222747 12793 external_mini_cluster.cc:1453] Setting key d20978e3700b2512875eff3ad53be80a
I20250109 21:03:00.223706 14391 heartbeater.cc:346] Connected to a master server at 127.12.126.126:42839
I20250109 21:03:00.223850 14391 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:00.224308 14391 heartbeater.cc:510] Master 127.12.126.126:42839 requested a full tablet report, sending...
I20250109 21:03:00.225265 13908 ts_manager.cc:194] Registered new tserver with Master: 3a09d6e0acc04084a3471c19e3b2d8d4 (127.12.126.67:43645)
I20250109 21:03:00.225962 13908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:47985
I20250109 21:03:00.231527 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.225986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:50889 (local address 127.12.126.126:42839)
0109 21:03:00.226102 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.226107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.226292 (+ 185us) server_negotiation.cc:407] Connection header received
0109 21:03:00.226403 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.226406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.226451 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.226529 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:00.227055 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.227761 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.228303 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.228427 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.229131 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:00.229145 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:00.229148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:00.229168 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:00.230473 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:00.230758 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:00.230760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:00.230761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:00.230794 (+ 33us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:00.230993 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:00.230995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:00.230996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:00.231110 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:00.231177 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.231341 (+ 164us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.231419 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20250109 21:03:00.231530 14397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.225944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42839 (local address 127.12.126.67:50889)
0109 21:03:00.226214 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:00.226226 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:00.226320 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:00.226558 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:00.226560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:00.226708 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:00.226947 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.226954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.227870 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.227872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:00.228222 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.228229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.228311 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.228771 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:00.228782 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:00.229035 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:00.230554 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:00.230556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:00.230557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:00.230694 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:00.230893 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:00.230895 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:00.230897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:00.230938 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:00.231171 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:00.231173 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:00.231238 (+ 65us) client_negotiation.cc:769] Sending connection context
0109 21:03:00.231311 (+ 73us) client_negotiation.cc:240] Negotiation successful
0109 21:03:00.231399 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":102,"threads_started":1}
I20250109 21:03:00.236155 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
Jan 09 21:03:00 dist-test-slave-g0zb krb5kdc[13863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456580, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 09 21:03:00 dist-test-slave-g0zb krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456580, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:00.259321 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.251511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51632 (local address 127.12.126.126:42839)
0109 21:03:00.251706 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.251712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.251730 (+ 18us) server_negotiation.cc:407] Connection header received
0109 21:03:00.251831 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.251835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.251892 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.252101 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:00.253000 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.253534 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.254149 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.254359 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.256535 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:00.256554 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:00.256557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:00.256581 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:00.257848 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:00.258245 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:00.258251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:00.258253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:00.258299 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:00.258507 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:00.258510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:00.258512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:00.258653 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:00.258772 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.259047 (+ 275us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.259162 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250109 21:03:00.262048 13908 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:03:00.264724 13908 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:03:00.277134 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.272418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:43019 (local address 127.0.0.1:47826)
0109 21:03:00.273456 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:00.273469 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:00.273561 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:00.274283 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:00.274286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:00.274306 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:00.274536 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.274548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.275630 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.275633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:00.276675 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.276688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.276796 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.276815 (+ 19us) client_negotiation.cc:769] Sending connection context
0109 21:03:00.276876 (+ 61us) client_negotiation.cc:240] Negotiation successful
0109 21:03:00.276961 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"thread_start_us":84,"threads_started":1}
I20250109 21:03:00.277279 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.272688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:39535 (local address 127.0.0.1:43656)
0109 21:03:00.273716 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:00.273727 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:00.273801 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:00.274475 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:00.274480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:00.274498 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:00.274704 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.274713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.276114 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.276117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:00.276957 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.276968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.277060 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.277075 (+ 15us) client_negotiation.cc:769] Sending connection context
0109 21:03:00.277119 (+ 44us) client_negotiation.cc:240] Negotiation successful
0109 21:03:00.277175 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"thread_start_us":89,"threads_started":1}
I20250109 21:03:00.277743 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.272486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40702 (local address 127.12.126.67:43645)
0109 21:03:00.273460 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.273469 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.273495 (+ 26us) server_negotiation.cc:407] Connection header received
0109 21:03:00.273603 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.273611 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.273802 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.273956 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:00.274408 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.275538 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.276771 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.277317 (+ 546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.277433 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.277519 (+ 86us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.277611 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844,"thread_start_us":115,"threads_started":1}
I20250109 21:03:00.277140 14409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.272329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:43645 (local address 127.0.0.1:40702)
0109 21:03:00.272953 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:00.272992 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:00.273116 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:00.273960 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:00.273965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:00.273984 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:00.274268 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.274278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.275664 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.275667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:00.276612 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.276622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.276726 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.276779 (+ 53us) client_negotiation.cc:769] Sending connection context
0109 21:03:00.276868 (+ 89us) client_negotiation.cc:240] Negotiation successful
0109 21:03:00.276957 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":60,"threads_started":1}
I20250109 21:03:00.278160 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.273054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43656 (local address 127.12.126.65:39535)
0109 21:03:00.273353 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.273362 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.273905 (+ 543us) server_negotiation.cc:407] Connection header received
0109 21:03:00.273986 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.273989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.274143 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.274304 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:00.274858 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.275991 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.277232 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.277751 (+ 519us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.277872 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.277940 (+ 68us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.278034 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20250109 21:03:00.278668 14413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.273063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47826 (local address 127.12.126.66:43019)
0109 21:03:00.273839 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.273846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.273869 (+ 23us) server_negotiation.cc:407] Connection header received
0109 21:03:00.273965 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.273970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.274143 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.274304 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:00.274659 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.275524 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.277771 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.278346 (+ 575us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.278443 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.278502 (+ 59us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.278567 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":92,"threads_started":1}
I20250109 21:03:00.280611 14045 tablet_service.cc:1467] Processing CreateTablet for tablet dcdf048726ef40e78c7c1dc6aed27760 (DEFAULT_TABLE table=test-table [id=cf685fab877d47beb60709c5f808d63a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:00.280985 14185 tablet_service.cc:1467] Processing CreateTablet for tablet dcdf048726ef40e78c7c1dc6aed27760 (DEFAULT_TABLE table=test-table [id=cf685fab877d47beb60709c5f808d63a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:00.280985 14325 tablet_service.cc:1467] Processing CreateTablet for tablet dcdf048726ef40e78c7c1dc6aed27760 (DEFAULT_TABLE table=test-table [id=cf685fab877d47beb60709c5f808d63a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:00.281510 14045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcdf048726ef40e78c7c1dc6aed27760. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:00.281848 14185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcdf048726ef40e78c7c1dc6aed27760. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:00.281848 14325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcdf048726ef40e78c7c1dc6aed27760. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:00.287024 14414 tablet_bootstrap.cc:492] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53: Bootstrap starting.
I20250109 21:03:00.288411 14415 tablet_bootstrap.cc:492] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4: Bootstrap starting.
I20250109 21:03:00.289104 14416 tablet_bootstrap.cc:492] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736: Bootstrap starting.
I20250109 21:03:00.290714 14416 tablet_bootstrap.cc:654] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:00.290726 14414 tablet_bootstrap.cc:654] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:00.290794 14415 tablet_bootstrap.cc:654] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:00.291486 14416 log.cc:826] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:00.291584 14415 log.cc:826] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:00.291589 14414 log.cc:826] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:00.293269 14416 tablet_bootstrap.cc:492] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736: No bootstrap required, opened a new log
I20250109 21:03:00.293346 14415 tablet_bootstrap.cc:492] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4: No bootstrap required, opened a new log
I20250109 21:03:00.293349 14414 tablet_bootstrap.cc:492] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53: No bootstrap required, opened a new log
I20250109 21:03:00.293452 14416 ts_tablet_manager.cc:1397] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:03:00.293504 14414 ts_tablet_manager.cc:1397] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20250109 21:03:00.293510 14415 ts_tablet_manager.cc:1397] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:03:00.295979 14416 raft_consensus.cc:357] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } }
I20250109 21:03:00.296164 14416 raft_consensus.cc:383] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:00.296259 14416 raft_consensus.cc:738] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5fd5416a3094aefbbb4d2cc835c3736, State: Initialized, Role: FOLLOWER
I20250109 21:03:00.296700 14416 consensus_queue.cc:260] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } }
I20250109 21:03:00.296653 14415 raft_consensus.cc:357] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } }
I20250109 21:03:00.296653 14414 raft_consensus.cc:357] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } }
I20250109 21:03:00.296840 14414 raft_consensus.cc:383] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:00.296840 14415 raft_consensus.cc:383] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:00.296907 14415 raft_consensus.cc:738] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09d6e0acc04084a3471c19e3b2d8d4, State: Initialized, Role: FOLLOWER
I20250109 21:03:00.296927 14414 raft_consensus.cc:738] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9c542d489c14eaabf0bf9b425dbaf53, State: Initialized, Role: FOLLOWER
I20250109 21:03:00.297336 14415 consensus_queue.cc:260] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } }
I20250109 21:03:00.297335 14414 consensus_queue.cc:260] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } }
I20250109 21:03:00.297426 14251 heartbeater.cc:502] Master 127.12.126.126:42839 was elected leader, sending a full tablet report...
I20250109 21:03:00.297586 14416 ts_tablet_manager.cc:1428] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250109 21:03:00.298113 14111 heartbeater.cc:502] Master 127.12.126.126:42839 was elected leader, sending a full tablet report...
I20250109 21:03:00.298185 14415 ts_tablet_manager.cc:1428] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:03:00.298185 14414 ts_tablet_manager.cc:1428] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20250109 21:03:00.298493 14391 heartbeater.cc:502] Master 127.12.126.126:42839 was elected leader, sending a full tablet report...
W20250109 21:03:00.311103 14112 tablet.cc:2367] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250109 21:03:00.471741 14392 tablet.cc:2367] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:03:00.511765 14421 raft_consensus.cc:491] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:03:00.511972 14421 raft_consensus.cc:513] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } }
I20250109 21:03:00.513110 14421 leader_election.cc:290] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a09d6e0acc04084a3471c19e3b2d8d4 (127.12.126.67:43645), e5fd5416a3094aefbbb4d2cc835c3736 (127.12.126.66:43019)
W20250109 21:03:00.514330 14252 tablet.cc:2367] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:03:00.516407 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.513355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:43645 (local address 127.12.126.65:53803)
0109 21:03:00.513704 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:00.513719 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:00.513862 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:00.514143 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:00.514146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:00.514182 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:00.514492 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.514500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.515348 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.515352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:00.515985 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.515995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.516093 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.516113 (+ 20us) client_negotiation.cc:769] Sending connection context
0109 21:03:00.516165 (+ 52us) client_negotiation.cc:240] Negotiation successful
0109 21:03:00.516237 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"spinlock_wait_cycles":46976,"thread_start_us":77,"threads_started":1}
I20250109 21:03:00.516876 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.513460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:53803 (local address 127.12.126.67:43645)
0109 21:03:00.513628 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.513633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.513800 (+ 167us) server_negotiation.cc:407] Connection header received
0109 21:03:00.513975 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.513978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.514033 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.514120 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:00.514599 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.515226 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.516107 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.516613 (+ 506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.516653 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.516704 (+ 51us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.516759 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20250109 21:03:00.517241 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.513395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:43019 (local address 127.12.126.65:51211)
0109 21:03:00.513736 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:00.513764 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:00.513862 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:00.514143 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:00.514146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:00.514167 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:00.514484 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.514493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.515536 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.515542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:00.516165 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.516176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.516911 (+ 735us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.516931 (+ 20us) client_negotiation.cc:769] Sending connection context
0109 21:03:00.517035 (+ 104us) client_negotiation.cc:240] Negotiation successful
0109 21:03:00.517098 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"spinlock_wait_cycles":7808,"thread_start_us":99,"threads_started":1}
I20250109 21:03:00.517241 14413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.513520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:51211 (local address 127.12.126.66:43019)
0109 21:03:00.513683 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.513688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.513830 (+ 142us) server_negotiation.cc:407] Connection header received
0109 21:03:00.513975 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.513978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.514033 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.514120 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:00.514882 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.515409 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.516339 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.516810 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.516843 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.517076 (+ 233us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.517144 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20250109 21:03:00.517539 14344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdf048726ef40e78c7c1dc6aed27760" candidate_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" is_pre_election: true
I20250109 21:03:00.517792 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdf048726ef40e78c7c1dc6aed27760" candidate_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" is_pre_election: true
I20250109 21:03:00.517836 14344 raft_consensus.cc:2463] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9c542d489c14eaabf0bf9b425dbaf53 in term 0.
I20250109 21:03:00.518064 14205 raft_consensus.cc:2463] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9c542d489c14eaabf0bf9b425dbaf53 in term 0.
I20250109 21:03:00.518311 13997 leader_election.cc:304] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09d6e0acc04084a3471c19e3b2d8d4, f9c542d489c14eaabf0bf9b425dbaf53; no voters:
I20250109 21:03:00.518564 14421 raft_consensus.cc:2798] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:03:00.518635 14421 raft_consensus.cc:491] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:03:00.518666 14421 raft_consensus.cc:3054] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:00.519658 14421 raft_consensus.cc:513] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } }
I20250109 21:03:00.520010 14421 leader_election.cc:290] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [CANDIDATE]: Term 1 election: Requested vote from peers 3a09d6e0acc04084a3471c19e3b2d8d4 (127.12.126.67:43645), e5fd5416a3094aefbbb4d2cc835c3736 (127.12.126.66:43019)
I20250109 21:03:00.520361 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdf048726ef40e78c7c1dc6aed27760" candidate_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5fd5416a3094aefbbb4d2cc835c3736"
I20250109 21:03:00.520382 14344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdf048726ef40e78c7c1dc6aed27760" candidate_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4"
I20250109 21:03:00.520479 14344 raft_consensus.cc:3054] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:00.520491 14205 raft_consensus.cc:3054] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:00.521395 14344 raft_consensus.cc:2463] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9c542d489c14eaabf0bf9b425dbaf53 in term 1.
I20250109 21:03:00.521407 14205 raft_consensus.cc:2463] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9c542d489c14eaabf0bf9b425dbaf53 in term 1.
I20250109 21:03:00.521696 13997 leader_election.cc:304] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a09d6e0acc04084a3471c19e3b2d8d4, e5fd5416a3094aefbbb4d2cc835c3736, f9c542d489c14eaabf0bf9b425dbaf53; no voters:
I20250109 21:03:00.521840 14421 raft_consensus.cc:2798] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:00.522009 14421 raft_consensus.cc:695] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [term 1 LEADER]: Becoming Leader. State: Replica: f9c542d489c14eaabf0bf9b425dbaf53, State: Running, Role: LEADER
I20250109 21:03:00.522308 14421 consensus_queue.cc:237] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } }
I20250109 21:03:00.525434 13906 catalog_manager.cc:5526] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9c542d489c14eaabf0bf9b425dbaf53 (127.12.126.65). New cstate: current_term: 1 leader_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } health_report { overall_health: HEALTHY } } }
I20250109 21:03:00.573009 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.569755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43666 (local address 127.12.126.65:39535)
0109 21:03:00.569883 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.569887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.569901 (+ 14us) server_negotiation.cc:407] Connection header received
0109 21:03:00.570157 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.570161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.570235 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.570341 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:00.570716 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.571297 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.571941 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.572097 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.572158 (+ 61us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:00.572637 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:00.572738 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.572843 (+ 105us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.572894 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250109 21:03:00.578656 14344 raft_consensus.cc:1270] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 1 FOLLOWER]: Refusing update from remote peer f9c542d489c14eaabf0bf9b425dbaf53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:00.578661 14205 raft_consensus.cc:1270] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736 [term 1 FOLLOWER]: Refusing update from remote peer f9c542d489c14eaabf0bf9b425dbaf53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:00.579367 14425 consensus_queue.cc:1035] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:00.579551 14421 consensus_queue.cc:1035] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:00.588798 14429 mvcc.cc:204] Tried to move back new op lower bound from 7112526154043105280 to 7112526153821782016. Current Snapshot: MvccSnapshot[applied={T|T < 7112526154043105280 or (T in {7112526154043105280})}]
I20250109 21:03:00.598366 13908 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51632:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:00.598528 13908 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51632:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:00.600978 13908 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8: Sending DeleteTablet for 3 replicas of tablet dcdf048726ef40e78c7c1dc6aed27760
I20250109 21:03:00.601977 14045 tablet_service.cc:1514] Processing DeleteTablet for tablet dcdf048726ef40e78c7c1dc6aed27760 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:00 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:43656
I20250109 21:03:00.602044 14185 tablet_service.cc:1514] Processing DeleteTablet for tablet dcdf048726ef40e78c7c1dc6aed27760 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:00 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:47826
I20250109 21:03:00.602519 14437 tablet_replica.cc:331] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53: stopping tablet replica
I20250109 21:03:00.602701 14325 tablet_service.cc:1514] Processing DeleteTablet for tablet dcdf048726ef40e78c7c1dc6aed27760 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:00 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:40702
I20250109 21:03:00.602839 14437 raft_consensus.cc:2238] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:03:00.602855 13908 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 21:03:00.603272 14437 raft_consensus.cc:2267] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 21:03:00.603284 13908 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:03:00.603320 14439 tablet_replica.cc:331] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4: stopping tablet replica
I20250109 21:03:00.603739 14439 raft_consensus.cc:2238] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:00.604095 14439 raft_consensus.cc:2267] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:00.605247 14437 ts_tablet_manager.cc:1905] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:00.605707 14439 ts_tablet_manager.cc:1905] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:00.605993 14438 tablet_replica.cc:331] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736: stopping tablet replica
I20250109 21:03:00.606318 14438 raft_consensus.cc:2238] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:00.608605 14045 tablet_service.cc:1467] Processing CreateTablet for tablet 9765560b15ac403bafc63e00c65d6140 (DEFAULT_TABLE table=test-table [id=cd92e8db845d47df9863afc2ecb1e8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:00.608860 14045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9765560b15ac403bafc63e00c65d6140. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:00.608645 14185 tablet_service.cc:1467] Processing CreateTablet for tablet 9765560b15ac403bafc63e00c65d6140 (DEFAULT_TABLE table=test-table [id=cd92e8db845d47df9863afc2ecb1e8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:00.609119 14185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9765560b15ac403bafc63e00c65d6140. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:00.609125 14325 tablet_service.cc:1467] Processing CreateTablet for tablet 9765560b15ac403bafc63e00c65d6140 (DEFAULT_TABLE table=test-table [id=cd92e8db845d47df9863afc2ecb1e8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:00.609380 14325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9765560b15ac403bafc63e00c65d6140. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:00.611480 14414 tablet_bootstrap.cc:492] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53: Bootstrap starting.
I20250109 21:03:00.611567 14415 tablet_bootstrap.cc:492] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4: Bootstrap starting.
I20250109 21:03:00.612555 14415 tablet_bootstrap.cc:654] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:00.612586 14414 tablet_bootstrap.cc:654] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:00.613175 14416 tablet_bootstrap.cc:492] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736: Bootstrap starting.
I20250109 21:03:00.613924 14415 tablet_bootstrap.cc:492] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4: No bootstrap required, opened a new log
I20250109 21:03:00.614019 14415 ts_tablet_manager.cc:1397] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:00.614148 14416 tablet_bootstrap.cc:654] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:00.614466 14415 raft_consensus.cc:357] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } }
I20250109 21:03:00.614562 14415 raft_consensus.cc:383] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:00.614599 14415 raft_consensus.cc:738] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09d6e0acc04084a3471c19e3b2d8d4, State: Initialized, Role: FOLLOWER
I20250109 21:03:00.614751 14415 consensus_queue.cc:260] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } }
I20250109 21:03:00.615064 14415 ts_tablet_manager.cc:1428] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 21:03:00.615661 14438 raft_consensus.cc:2267] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:00.616807 14437 ts_tablet_manager.cc:1918] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:03:00.616948 14437 log.cc:1198] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/wal/wals/dcdf048726ef40e78c7c1dc6aed27760
I20250109 21:03:00.617403 14437 ts_tablet_manager.cc:1939] T dcdf048726ef40e78c7c1dc6aed27760 P f9c542d489c14eaabf0bf9b425dbaf53: Deleting consensus metadata
I20250109 21:03:00.617786 14439 ts_tablet_manager.cc:1918] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:03:00.617870 14439 log.cc:1198] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/dcdf048726ef40e78c7c1dc6aed27760
I20250109 21:03:00.618866 14439 ts_tablet_manager.cc:1939] T dcdf048726ef40e78c7c1dc6aed27760 P 3a09d6e0acc04084a3471c19e3b2d8d4: Deleting consensus metadata
I20250109 21:03:00.618927 14416 tablet_bootstrap.cc:492] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736: No bootstrap required, opened a new log
I20250109 21:03:00.619025 14416 ts_tablet_manager.cc:1397] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20250109 21:03:00.619365 14414 tablet_bootstrap.cc:492] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53: No bootstrap required, opened a new log
I20250109 21:03:00.619439 14414 ts_tablet_manager.cc:1397] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20250109 21:03:00.619498 14416 raft_consensus.cc:357] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } }
I20250109 21:03:00.619612 14416 raft_consensus.cc:383] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:00.619647 14416 raft_consensus.cc:738] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5fd5416a3094aefbbb4d2cc835c3736, State: Initialized, Role: FOLLOWER
I20250109 21:03:00.619642 13894 catalog_manager.cc:4872] TS f9c542d489c14eaabf0bf9b425dbaf53 (127.12.126.65:39535): tablet dcdf048726ef40e78c7c1dc6aed27760 (table test-table [id=cf685fab877d47beb60709c5f808d63a]) successfully deleted
I20250109 21:03:00.619781 14416 consensus_queue.cc:260] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } }
I20250109 21:03:00.619874 14414 raft_consensus.cc:357] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } }
I20250109 21:03:00.619973 14414 raft_consensus.cc:383] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:00.620003 14414 raft_consensus.cc:738] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9c542d489c14eaabf0bf9b425dbaf53, State: Initialized, Role: FOLLOWER
I20250109 21:03:00.620038 14416 ts_tablet_manager.cc:1428] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250109 21:03:00.620116 14414 consensus_queue.cc:260] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } }
I20250109 21:03:00.620340 14414 ts_tablet_manager.cc:1428] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 21:03:00.622022 13893 catalog_manager.cc:4872] TS 3a09d6e0acc04084a3471c19e3b2d8d4 (127.12.126.67:43645): tablet dcdf048726ef40e78c7c1dc6aed27760 (table test-table [id=cf685fab877d47beb60709c5f808d63a]) successfully deleted
I20250109 21:03:00.623597 14438 ts_tablet_manager.cc:1905] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:00.623992 14422 raft_consensus.cc:491] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:03:00.624086 14422 raft_consensus.cc:513] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } }
I20250109 21:03:00.625139 14422 leader_election.cc:290] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9c542d489c14eaabf0bf9b425dbaf53 (127.12.126.65:39535), e5fd5416a3094aefbbb4d2cc835c3736 (127.12.126.66:43019)
I20250109 21:03:00.626107 14438 ts_tablet_manager.cc:1918] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:03:00.626215 14438 log.cc:1198] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/dcdf048726ef40e78c7c1dc6aed27760
I20250109 21:03:00.626552 14438 ts_tablet_manager.cc:1939] T dcdf048726ef40e78c7c1dc6aed27760 P e5fd5416a3094aefbbb4d2cc835c3736: Deleting consensus metadata
I20250109 21:03:00.627689 13892 catalog_manager.cc:4872] TS e5fd5416a3094aefbbb4d2cc835c3736 (127.12.126.66:43019): tablet dcdf048726ef40e78c7c1dc6aed27760 (table test-table [id=cf685fab877d47beb60709c5f808d63a]) successfully deleted
I20250109 21:03:00.629124 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.625311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:39535 (local address 127.12.126.67:51217)
0109 21:03:00.625560 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:00.625576 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:00.625662 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:00.626638 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:00.626640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:00.626659 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:00.626922 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.626931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.628172 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.628175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:00.628745 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.628752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.628863 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.628886 (+ 23us) client_negotiation.cc:769] Sending connection context
0109 21:03:00.628940 (+ 54us) client_negotiation.cc:240] Negotiation successful
0109 21:03:00.629000 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20250109 21:03:00.629608 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.625845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:51217 (local address 127.12.126.65:39535)
0109 21:03:00.625958 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.625962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.625976 (+ 14us) server_negotiation.cc:407] Connection header received
0109 21:03:00.626020 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.626025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.626084 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.626155 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:00.627252 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.628017 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.628863 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.629345 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.629386 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.629450 (+ 64us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.629526 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250109 21:03:00.630209 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.625388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:43019 (local address 127.12.126.67:51275)
0109 21:03:00.626020 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:00.626043 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:00.626184 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:00.626587 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:00.626897 (+ 310us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:00.626919 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:00.627153 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.627162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.628137 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.628141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:00.629124 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.629142 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.629886 (+ 744us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.629902 (+ 16us) client_negotiation.cc:769] Sending connection context
0109 21:03:00.630024 (+ 122us) client_negotiation.cc:240] Negotiation successful
0109 21:03:00.630082 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":108,"threads_started":1}
I20250109 21:03:00.630157 14065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9765560b15ac403bafc63e00c65d6140" candidate_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" is_pre_election: true
I20250109 21:03:00.630257 14413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.625750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:51275 (local address 127.12.126.66:43019)
0109 21:03:00.625865 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.625870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.626115 (+ 245us) server_negotiation.cc:407] Connection header received
0109 21:03:00.626315 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.626318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.626375 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.626426 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:00.627279 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.628017 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.629286 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.629772 (+ 486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.629806 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.630073 (+ 267us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.630144 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250109 21:03:00.630374 14065 raft_consensus.cc:2463] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a09d6e0acc04084a3471c19e3b2d8d4 in term 0.
I20250109 21:03:00.630860 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9765560b15ac403bafc63e00c65d6140" candidate_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" is_pre_election: true
I20250109 21:03:00.630892 14278 leader_election.cc:304] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09d6e0acc04084a3471c19e3b2d8d4, f9c542d489c14eaabf0bf9b425dbaf53; no voters:
I20250109 21:03:00.630980 14205 raft_consensus.cc:2463] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a09d6e0acc04084a3471c19e3b2d8d4 in term 0.
I20250109 21:03:00.631210 14422 raft_consensus.cc:2798] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:03:00.631278 14422 raft_consensus.cc:491] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:03:00.631323 14422 raft_consensus.cc:3054] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:00.632401 14422 raft_consensus.cc:513] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } }
I20250109 21:03:00.632792 14422 leader_election.cc:290] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [CANDIDATE]: Term 1 election: Requested vote from peers f9c542d489c14eaabf0bf9b425dbaf53 (127.12.126.65:39535), e5fd5416a3094aefbbb4d2cc835c3736 (127.12.126.66:43019)
I20250109 21:03:00.633088 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9765560b15ac403bafc63e00c65d6140" candidate_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5fd5416a3094aefbbb4d2cc835c3736"
I20250109 21:03:00.633088 14065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9765560b15ac403bafc63e00c65d6140" candidate_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9c542d489c14eaabf0bf9b425dbaf53"
I20250109 21:03:00.633189 14205 raft_consensus.cc:3054] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:00.633203 14065 raft_consensus.cc:3054] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:00.634250 14205 raft_consensus.cc:2463] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a09d6e0acc04084a3471c19e3b2d8d4 in term 1.
I20250109 21:03:00.634272 14065 raft_consensus.cc:2463] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a09d6e0acc04084a3471c19e3b2d8d4 in term 1.
I20250109 21:03:00.634552 14276 leader_election.cc:304] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09d6e0acc04084a3471c19e3b2d8d4, e5fd5416a3094aefbbb4d2cc835c3736; no voters:
I20250109 21:03:00.634734 14422 raft_consensus.cc:2798] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:00.634950 14422 raft_consensus.cc:695] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 1 LEADER]: Becoming Leader. State: Replica: 3a09d6e0acc04084a3471c19e3b2d8d4, State: Running, Role: LEADER
I20250109 21:03:00.635208 14422 consensus_queue.cc:237] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } }
I20250109 21:03:00.637298 13907 catalog_manager.cc:5526] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a09d6e0acc04084a3471c19e3b2d8d4 (127.12.126.67). New cstate: current_term: 1 leader_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } health_report { overall_health: HEALTHY } } }
Jan 09 21:03:00 dist-test-slave-g0zb krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456579, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:00.677122 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.668840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.12.126.126:42839)
0109 21:03:00.669005 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.669010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.669155 (+ 145us) server_negotiation.cc:407] Connection header received
0109 21:03:00.669341 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.669343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.669381 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.669455 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:00.670268 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.670762 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.671707 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.671838 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.674135 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:00.674152 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:00.674154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:00.674175 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:00.675593 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:00.676023 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:00.676026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:00.676030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:00.676067 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:00.676270 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:00.676273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:00.676275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:00.676414 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:00.676517 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.676784 (+ 267us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.676955 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":67}
I20250109 21:03:00.677124 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.668736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42839 (local address 127.0.0.1:51646)
0109 21:03:00.669065 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:00.669079 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:00.669224 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:00.669477 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:00.669480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:00.669944 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:00.670138 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.670145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.670908 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.670913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:00.671576 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.671593 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.671726 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.672351 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:00.672367 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:00.674012 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:00.675699 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:00.675708 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:00.675714 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:00.675933 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:00.676143 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:00.676146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:00.676149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:00.676197 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:00.676526 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:00.676534 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:00.676633 (+ 99us) client_negotiation.cc:769] Sending connection context
0109 21:03:00.676732 (+ 99us) client_negotiation.cc:240] Negotiation successful
0109 21:03:00.676887 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":123,"threads_started":1}
I20250109 21:03:00.681939 13907 catalog_manager.cc:1909] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:51646:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250109 21:03:00.682852 13907 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:03:00.688061 14045 tablet_service.cc:1467] Processing CreateTablet for tablet 3acbe333eee1444da41fbc9a491f940d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e2c18a0ff8994a9ea758bbb2d8c84240]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250109 21:03:00.688146 14325 tablet_service.cc:1467] Processing CreateTablet for tablet 3acbe333eee1444da41fbc9a491f940d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e2c18a0ff8994a9ea758bbb2d8c84240]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250109 21:03:00.688289 14185 tablet_service.cc:1467] Processing CreateTablet for tablet 3acbe333eee1444da41fbc9a491f940d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e2c18a0ff8994a9ea758bbb2d8c84240]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250109 21:03:00.688517 14325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3acbe333eee1444da41fbc9a491f940d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:00.688522 14045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3acbe333eee1444da41fbc9a491f940d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:00.688782 14185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3acbe333eee1444da41fbc9a491f940d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:00.691251 14414 tablet_bootstrap.cc:492] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53: Bootstrap starting.
I20250109 21:03:00.691303 14415 tablet_bootstrap.cc:492] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4: Bootstrap starting.
I20250109 21:03:00.692272 14415 tablet_bootstrap.cc:654] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:00.692639 14414 tablet_bootstrap.cc:654] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:00.693694 14416 tablet_bootstrap.cc:492] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736: Bootstrap starting.
I20250109 21:03:00.693818 14415 tablet_bootstrap.cc:492] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4: No bootstrap required, opened a new log
I20250109 21:03:00.693888 14415 ts_tablet_manager.cc:1397] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250109 21:03:00.694365 14415 raft_consensus.cc:357] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } }
I20250109 21:03:00.694470 14415 raft_consensus.cc:383] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:00.694501 14415 raft_consensus.cc:738] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a09d6e0acc04084a3471c19e3b2d8d4, State: Initialized, Role: FOLLOWER
I20250109 21:03:00.694625 14415 consensus_queue.cc:260] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } }
I20250109 21:03:00.694749 14416 tablet_bootstrap.cc:654] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:00.694934 14422 tablet_replica.cc:440] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } }
I20250109 21:03:00.695048 14422 tablet_replica.cc:443] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:00.695441 14414 tablet_bootstrap.cc:492] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53: No bootstrap required, opened a new log
I20250109 21:03:00.695546 14414 ts_tablet_manager.cc:1397] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20250109 21:03:00.695977 14414 raft_consensus.cc:357] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } }
I20250109 21:03:00.696086 14414 raft_consensus.cc:383] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:00.696118 14414 raft_consensus.cc:738] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9c542d489c14eaabf0bf9b425dbaf53, State: Initialized, Role: FOLLOWER
I20250109 21:03:00.696252 14414 consensus_queue.cc:260] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } }
I20250109 21:03:00.696524 14414 ts_tablet_manager.cc:1428] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 21:03:00.696761 14421 tablet_replica.cc:440] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } }
I20250109 21:03:00.696883 14421 tablet_replica.cc:443] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:00.697981 14415 ts_tablet_manager.cc:1428] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250109 21:03:00.699018 14416 tablet_bootstrap.cc:492] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736: No bootstrap required, opened a new log
I20250109 21:03:00.699118 14416 ts_tablet_manager.cc:1397] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20250109 21:03:00.699656 14416 raft_consensus.cc:357] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } }
I20250109 21:03:00.699774 14416 raft_consensus.cc:383] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:00.699810 14416 raft_consensus.cc:738] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5fd5416a3094aefbbb4d2cc835c3736, State: Initialized, Role: FOLLOWER
I20250109 21:03:00.699957 14416 consensus_queue.cc:260] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } }
I20250109 21:03:00.700225 14420 tablet_replica.cc:440] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } }
I20250109 21:03:00.700353 14420 tablet_replica.cc:443] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:00.700239 14416 ts_tablet_manager.cc:1428] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 21:03:00.754355 14422 raft_consensus.cc:491] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:03:00.754508 14422 raft_consensus.cc:513] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } }
I20250109 21:03:00.754969 14422 leader_election.cc:290] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9c542d489c14eaabf0bf9b425dbaf53 (127.12.126.65:39535), e5fd5416a3094aefbbb4d2cc835c3736 (127.12.126.66:43019)
I20250109 21:03:00.755492 14065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acbe333eee1444da41fbc9a491f940d" candidate_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" is_pre_election: true
I20250109 21:03:00.755527 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acbe333eee1444da41fbc9a491f940d" candidate_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" is_pre_election: true
I20250109 21:03:00.755631 14065 raft_consensus.cc:2463] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a09d6e0acc04084a3471c19e3b2d8d4 in term 0.
I20250109 21:03:00.755702 14205 raft_consensus.cc:2463] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a09d6e0acc04084a3471c19e3b2d8d4 in term 0.
I20250109 21:03:00.755929 14278 leader_election.cc:304] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09d6e0acc04084a3471c19e3b2d8d4, f9c542d489c14eaabf0bf9b425dbaf53; no voters:
I20250109 21:03:00.756117 14422 raft_consensus.cc:2798] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:03:00.756177 14422 raft_consensus.cc:491] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:03:00.756230 14422 raft_consensus.cc:3054] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:00.757172 14422 raft_consensus.cc:513] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } }
I20250109 21:03:00.757532 14422 leader_election.cc:290] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [CANDIDATE]: Term 1 election: Requested vote from peers f9c542d489c14eaabf0bf9b425dbaf53 (127.12.126.65:39535), e5fd5416a3094aefbbb4d2cc835c3736 (127.12.126.66:43019)
I20250109 21:03:00.757894 14065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acbe333eee1444da41fbc9a491f940d" candidate_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9c542d489c14eaabf0bf9b425dbaf53"
I20250109 21:03:00.757915 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acbe333eee1444da41fbc9a491f940d" candidate_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5fd5416a3094aefbbb4d2cc835c3736"
I20250109 21:03:00.758026 14065 raft_consensus.cc:3054] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:00.758031 14205 raft_consensus.cc:3054] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:00.759133 14205 raft_consensus.cc:2463] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a09d6e0acc04084a3471c19e3b2d8d4 in term 1.
I20250109 21:03:00.759248 14065 raft_consensus.cc:2463] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a09d6e0acc04084a3471c19e3b2d8d4 in term 1.
I20250109 21:03:00.759490 14276 leader_election.cc:304] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a09d6e0acc04084a3471c19e3b2d8d4, e5fd5416a3094aefbbb4d2cc835c3736; no voters:
I20250109 21:03:00.759716 14422 raft_consensus.cc:2798] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:00.759788 14422 raft_consensus.cc:695] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 1 LEADER]: Becoming Leader. State: Replica: 3a09d6e0acc04084a3471c19e3b2d8d4, State: Running, Role: LEADER
I20250109 21:03:00.759928 14422 consensus_queue.cc:237] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } }
I20250109 21:03:00.760609 14444 tablet_replica.cc:440] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4: TxnStatusTablet state changed. Reason: New leader 3a09d6e0acc04084a3471c19e3b2d8d4. Latest consensus state: current_term: 1 leader_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } }
I20250109 21:03:00.760704 14444 tablet_replica.cc:443] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4: This TxnStatusTablet replica's current role is: LEADER
I20250109 21:03:00.760962 14456 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250109 21:03:00.761085 14456 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250109 21:03:00.761919 13908 catalog_manager.cc:5526] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a09d6e0acc04084a3471c19e3b2d8d4 (127.12.126.67). New cstate: current_term: 1 leader_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } health_report { overall_health: HEALTHY } } }
I20250109 21:03:00.785010 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.780939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40708 (local address 127.12.126.67:43645)
0109 21:03:00.781079 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.781084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.781218 (+ 134us) server_negotiation.cc:407] Connection header received
0109 21:03:00.781419 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.781422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.781491 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.781607 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:00.782024 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.782869 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.783781 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.783984 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.784050 (+ 66us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:00.784609 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:00.784727 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.784817 (+ 90us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.784874 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250109 21:03:00.785010 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.780853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:43645 (local address 127.0.0.1:40708)
0109 21:03:00.781141 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:00.781156 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:00.781269 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:00.781625 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:00.781628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:00.781636 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0109 21:03:00.781887 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.781893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.782976 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.782979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:00.783665 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.783675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.783786 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.783831 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0109 21:03:00.784710 (+ 879us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0109 21:03:00.784721 (+ 11us) client_negotiation.cc:769] Sending connection context
0109 21:03:00.784787 (+ 66us) client_negotiation.cc:240] Negotiation successful
0109 21:03:00.784859 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20250109 21:03:00.789108 14205 raft_consensus.cc:1270] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736 [term 1 FOLLOWER]: Refusing update from remote peer 3a09d6e0acc04084a3471c19e3b2d8d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:00.789758 14065 raft_consensus.cc:1270] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53 [term 1 FOLLOWER]: Refusing update from remote peer 3a09d6e0acc04084a3471c19e3b2d8d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:00.790397 14422 consensus_queue.cc:1035] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:00.790575 14444 consensus_queue.cc:1035] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:00.792100 14420 tablet_replica.cc:440] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736: TxnStatusTablet state changed. Reason: New leader 3a09d6e0acc04084a3471c19e3b2d8d4. Latest consensus state: current_term: 1 leader_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } }
I20250109 21:03:00.792218 14420 tablet_replica.cc:443] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:00.793335 14421 tablet_replica.cc:440] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53: TxnStatusTablet state changed. Reason: New leader 3a09d6e0acc04084a3471c19e3b2d8d4. Latest consensus state: current_term: 1 leader_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } }
I20250109 21:03:00.793448 14421 tablet_replica.cc:443] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:00.795578 14444 tablet_replica.cc:440] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } }
I20250109 21:03:00.795625 14422 tablet_replica.cc:440] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } }
I20250109 21:03:00.795696 14422 tablet_replica.cc:443] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4: This TxnStatusTablet replica's current role is: LEADER
I20250109 21:03:00.796473 14420 tablet_replica.cc:440] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } }
I20250109 21:03:00.796612 14420 tablet_replica.cc:443] T 3acbe333eee1444da41fbc9a491f940d P e5fd5416a3094aefbbb4d2cc835c3736: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:00.796610 14421 tablet_replica.cc:440] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 } } peers { permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 } } peers { permanent_uuid: "3a09d6e0acc04084a3471c19e3b2d8d4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43645 } } }
I20250109 21:03:00.796708 14421 tablet_replica.cc:443] T 3acbe333eee1444da41fbc9a491f940d P f9c542d489c14eaabf0bf9b425dbaf53: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:00.797685 14428 mvcc.cc:204] Tried to move back new op lower bound from 7112526154905591808 to 7112526154793992192. Current Snapshot: MvccSnapshot[applied={T|T < 7112526154905591808 or (T in {7112526154905591808})}]
I20250109 21:03:00.795694 14444 tablet_replica.cc:443] T 3acbe333eee1444da41fbc9a491f940d P 3a09d6e0acc04084a3471c19e3b2d8d4: This TxnStatusTablet replica's current role is: LEADER
I20250109 21:03:00.804234 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.801384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40712 (local address 127.12.126.67:43645)
0109 21:03:00.801515 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.801519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.801535 (+ 16us) server_negotiation.cc:407] Connection header received
0109 21:03:00.801626 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.801628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.801680 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.801763 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:00.802178 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.802970 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.803574 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.803761 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.803809 (+ 48us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:00.803886 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:00.803961 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.804036 (+ 75us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.804103 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20250109 21:03:00.813509 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.810669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:54023 (local address 127.12.126.67:43645)
0109 21:03:00.810816 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.810821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.811042 (+ 221us) server_negotiation.cc:407] Connection header received
0109 21:03:00.811172 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.811175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.811213 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.811273 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:00.811583 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.812123 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.812895 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.813071 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.813112 (+ 41us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:00.813170 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:00.813232 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.813333 (+ 101us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.813368 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250109 21:03:00.813527 14464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.810647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:43645 (local address 127.12.126.67:54023)
0109 21:03:00.810964 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:00.810975 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:00.811077 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:00.811287 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:00.811290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:00.811296 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0109 21:03:00.811491 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.811497 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.812238 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.812241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:00.812793 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:00.812802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.812925 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.812969 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0109 21:03:00.813249 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0109 21:03:00.813257 (+ 8us) client_negotiation.cc:769] Sending connection context
0109 21:03:00.813321 (+ 64us) client_negotiation.cc:240] Negotiation successful
0109 21:03:00.813380 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":94,"threads_started":1}
I20250109 21:03:00.821185 14205 raft_consensus.cc:1270] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736 [term 1 FOLLOWER]: Refusing update from remote peer 3a09d6e0acc04084a3471c19e3b2d8d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:00.821277 14065 raft_consensus.cc:1270] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53 [term 1 FOLLOWER]: Refusing update from remote peer 3a09d6e0acc04084a3471c19e3b2d8d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:00.821650 14444 consensus_queue.cc:1035] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5fd5416a3094aefbbb4d2cc835c3736" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:00.821844 14422 consensus_queue.cc:1035] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9c542d489c14eaabf0bf9b425dbaf53" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:00.828106 14430 mvcc.cc:204] Tried to move back new op lower bound from 7112526155039784960 to 7112526154283905024. Current Snapshot: MvccSnapshot[applied={T|T < 7112526155039784960}]
I20250109 21:03:00.869697 13906 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51632:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:00.869848 13906 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51632:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:00.871924 13906 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 7e0fd249788a4a8ea6183baf6fb3dfb8: Sending DeleteTablet for 3 replicas of tablet 9765560b15ac403bafc63e00c65d6140
I20250109 21:03:00.872515 14045 tablet_service.cc:1514] Processing DeleteTablet for tablet 9765560b15ac403bafc63e00c65d6140 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:00 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:43656
I20250109 21:03:00.872651 14185 tablet_service.cc:1514] Processing DeleteTablet for tablet 9765560b15ac403bafc63e00c65d6140 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:00 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:47826
I20250109 21:03:00.872715 14437 tablet_replica.cc:331] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53: stopping tablet replica
I20250109 21:03:00.872831 14437 raft_consensus.cc:2238] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:00.872876 14438 tablet_replica.cc:331] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736: stopping tablet replica
I20250109 21:03:00.872951 14437 raft_consensus.cc:2267] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:00.872987 14438 raft_consensus.cc:2238] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:00.872967 14325 tablet_service.cc:1514] Processing DeleteTablet for tablet 9765560b15ac403bafc63e00c65d6140 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:00 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:40702
I20250109 21:03:00.873099 14438 raft_consensus.cc:2267] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:00.873126 14439 tablet_replica.cc:331] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4: stopping tablet replica
I20250109 21:03:00.873220 14439 raft_consensus.cc:2238] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:03:00.873548 14439 raft_consensus.cc:2267] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:00.873881 14438 ts_tablet_manager.cc:1905] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:00.874212 14439 ts_tablet_manager.cc:1905] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:00.874962 14437 ts_tablet_manager.cc:1905] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:00.876649 14438 ts_tablet_manager.cc:1918] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250109 21:03:00.876721 14438 log.cc:1198] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/9765560b15ac403bafc63e00c65d6140
I20250109 21:03:00.876754 14439 ts_tablet_manager.cc:1918] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250109 21:03:00.876832 14439 log.cc:1198] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/9765560b15ac403bafc63e00c65d6140
I20250109 21:03:00.876978 14438 ts_tablet_manager.cc:1939] T 9765560b15ac403bafc63e00c65d6140 P e5fd5416a3094aefbbb4d2cc835c3736: Deleting consensus metadata
I20250109 21:03:00.877101 14439 ts_tablet_manager.cc:1939] T 9765560b15ac403bafc63e00c65d6140 P 3a09d6e0acc04084a3471c19e3b2d8d4: Deleting consensus metadata
I20250109 21:03:00.877555 14437 ts_tablet_manager.cc:1918] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250109 21:03:00.877624 14437 log.cc:1198] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1736456573526088-12793-0/minicluster-data/ts-0/wal/wals/9765560b15ac403bafc63e00c65d6140
I20250109 21:03:00.877744 13892 catalog_manager.cc:4872] TS e5fd5416a3094aefbbb4d2cc835c3736 (127.12.126.66:43019): tablet 9765560b15ac403bafc63e00c65d6140 (table test-table [id=cd92e8db845d47df9863afc2ecb1e8f8]) successfully deleted
I20250109 21:03:00.877868 14437 ts_tablet_manager.cc:1939] T 9765560b15ac403bafc63e00c65d6140 P f9c542d489c14eaabf0bf9b425dbaf53: Deleting consensus metadata
I20250109 21:03:00.877938 13893 catalog_manager.cc:4872] TS 3a09d6e0acc04084a3471c19e3b2d8d4 (127.12.126.67:43645): tablet 9765560b15ac403bafc63e00c65d6140 (table test-table [id=cd92e8db845d47df9863afc2ecb1e8f8]) successfully deleted
I20250109 21:03:00.878474 13894 catalog_manager.cc:4872] TS f9c542d489c14eaabf0bf9b425dbaf53 (127.12.126.65:39535): tablet 9765560b15ac403bafc63e00c65d6140 (table test-table [id=cd92e8db845d47df9863afc2ecb1e8f8]) successfully deleted
Jan 09 21:03:00 dist-test-slave-g0zb krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456580, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.126.65@KRBTEST.COM
I20250109 21:03:00.887624 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.878663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43678 (local address 127.12.126.65:39535)
0109 21:03:00.878788 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.878793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.878978 (+ 185us) server_negotiation.cc:407] Connection header received
0109 21:03:00.879119 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.879122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.879168 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.879240 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:00.880019 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.880490 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.881142 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.881370 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.884270 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:00.884292 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:00.884304 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:00.884338 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:00.886044 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:00.886426 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:00.886432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:00.886435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:00.886493 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:00.886714 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:00.886717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:00.886719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:00.886966 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:00.887165 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.887331 (+ 166us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.887459 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20250109 21:03:00.888079 14105 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43678
I20250109 21:03:00.896595 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:00.889856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51652 (local address 127.12.126.126:42839)
0109 21:03:00.889985 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:00.889990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:00.890052 (+ 62us) server_negotiation.cc:407] Connection header received
0109 21:03:00.890187 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:00.890191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:00.890246 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:00.890318 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:00.891068 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.891593 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:00.892266 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:00.892508 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:00.893664 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:00.893682 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:00.893685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:00.893717 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:00.895239 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:00.895612 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:00.895616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:00.895617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:00.895659 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:00.895860 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:00.895864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:00.895866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:00.896003 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:00.896122 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:00.896321 (+ 199us) server_negotiation.cc:299] Negotiation successful
0109 21:03:00.896432 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20250109 21:03:00.897012 13906 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51652
I20250109 21:03:00.898151 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 13980
I20250109 21:03:00.907326 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 14120
I20250109 21:03:00.915139 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 14260
I20250109 21:03:00.921994 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 13876
2025-01-09T21:03:00Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2730 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:03:00 dist-test-slave-g0zb krb5kdc[14475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:03:00 dist-test-slave-g0zb krb5kdc[14475](info): set up 2 sockets
Jan 09 21:03:00 dist-test-slave-g0zb krb5kdc[14475](info): commencing operation
krb5kdc: starting...
W20250109 21:03:01.951303 12793 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:03:02 dist-test-slave-g0zb krb5kdc[14475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:03:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:03:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:03:02.069700 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:44583
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:44583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:02.175107 14488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:02.175371 14488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:02.175463 14488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:02.178633 14488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:02.178704 14488 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250109 21:03:02.178722 14488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:02.178751 14488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:02.178812 14488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:02.178849 14488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:03:02.183182 14488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:44583
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:44583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:02.184327 14488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:02.185180 14488 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:02.192291 14493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:02.193056 14496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:02.193707 14494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:02.194659 14488 server_base.cc:1034] running on GCE node
I20250109 21:03:02.195165 14488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:02.196070 14488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:02.197218 14488 hybrid_clock.cc:648] HybridClock initialized: now 1736456582197208 us; error 28 us; skew 500 ppm
Jan 09 21:03:02 dist-test-slave-g0zb krb5kdc[14475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456582, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:02.199935 14488 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:03:02.200814 14488 webserver.cc:458] Webserver started at http://127.12.126.126:39995/ using document root <none> and password file <none>
I20250109 21:03:02.201398 14488 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:02.201463 14488 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:02.201666 14488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:02.203466 14488 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "f27d668791c44620931035e3f3d7d7f0"
format_stamp: "Formatted at 2025-01-09 21:03:02 on dist-test-slave-g0zb"
server_key: "83f7ec362e3b1ca782502e2321bf1efe"
server_key_iv: "aa445545426bda5fa08bb62328b32fc6"
server_key_version: "encryptionkey@0"
I20250109 21:03:02.203938 14488 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "f27d668791c44620931035e3f3d7d7f0"
format_stamp: "Formatted at 2025-01-09 21:03:02 on dist-test-slave-g0zb"
server_key: "83f7ec362e3b1ca782502e2321bf1efe"
server_key_iv: "aa445545426bda5fa08bb62328b32fc6"
server_key_version: "encryptionkey@0"
I20250109 21:03:02.207219 14488 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:03:02.209360 14503 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:02.210400 14488 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:02.210533 14488 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "f27d668791c44620931035e3f3d7d7f0"
format_stamp: "Formatted at 2025-01-09 21:03:02 on dist-test-slave-g0zb"
server_key: "83f7ec362e3b1ca782502e2321bf1efe"
server_key_iv: "aa445545426bda5fa08bb62328b32fc6"
server_key_version: "encryptionkey@0"
I20250109 21:03:02.210623 14488 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:02.232596 14488 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:02.235441 14488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:02.235618 14488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:02.243185 14488 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:44583
I20250109 21:03:02.243194 14565 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:44583 every 8 connection(s)
I20250109 21:03:02.244177 14488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:03:02.244935 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 14488
I20250109 21:03:02.245030 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:03:02.245280 12793 external_mini_cluster.cc:1453] Setting key a9ddc61c0411368da87a04090b9534d4
I20250109 21:03:02.247062 14566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:02.252972 14566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0: Bootstrap starting.
Jan 09 21:03:02 dist-test-slave-g0zb krb5kdc[14475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:02.255532 14566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:02.256788 14566 log.cc:826] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:02.258292 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.246521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.12.126.126:44583)
0109 21:03:02.246932 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:02.246945 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:02.246984 (+ 39us) server_negotiation.cc:407] Connection header received
0109 21:03:02.247789 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:02.247813 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:02.248152 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:02.248563 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:02.249261 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.250499 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.251285 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.251578 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.253525 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:02.253582 (+ 57us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:02.253596 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:02.253633 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:02.255706 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:02.256108 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:02.256115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:02.256122 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:02.256249 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:02.256469 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:02.256472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:02.256474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:02.256892 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:02.257065 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:02.257281 (+ 216us) server_negotiation.cc:299] Negotiation successful
0109 21:03:02.257565 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":101,"threads_started":1}
I20250109 21:03:02.258749 14566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0: No bootstrap required, opened a new log
I20250109 21:03:02.261575 14566 raft_consensus.cc:357] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27d668791c44620931035e3f3d7d7f0" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 44583 } }
I20250109 21:03:02.261758 14566 raft_consensus.cc:383] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:02.261823 14566 raft_consensus.cc:738] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f27d668791c44620931035e3f3d7d7f0, State: Initialized, Role: FOLLOWER
I20250109 21:03:02.262223 14566 consensus_queue.cc:260] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27d668791c44620931035e3f3d7d7f0" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 44583 } }
I20250109 21:03:02.262344 14566 raft_consensus.cc:397] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:03:02.262409 14566 raft_consensus.cc:491] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:03:02.262518 14566 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:02.263343 14566 raft_consensus.cc:513] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27d668791c44620931035e3f3d7d7f0" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 44583 } }
I20250109 21:03:02.263713 14566 leader_election.cc:304] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f27d668791c44620931035e3f3d7d7f0; no voters:
I20250109 21:03:02.264051 14566 leader_election.cc:290] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:03:02.264153 14571 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:02.264413 14571 raft_consensus.cc:695] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0 [term 1 LEADER]: Becoming Leader. State: Replica: f27d668791c44620931035e3f3d7d7f0, State: Running, Role: LEADER
I20250109 21:03:02.264708 14571 consensus_queue.cc:237] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27d668791c44620931035e3f3d7d7f0" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 44583 } }
I20250109 21:03:02.265141 14566 sys_catalog.cc:564] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:02.266211 14572 sys_catalog.cc:455] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f27d668791c44620931035e3f3d7d7f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27d668791c44620931035e3f3d7d7f0" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 44583 } } }
I20250109 21:03:02.266330 14572 sys_catalog.cc:458] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:02.266585 14573 sys_catalog.cc:455] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f27d668791c44620931035e3f3d7d7f0. Latest consensus state: current_term: 1 leader_uuid: "f27d668791c44620931035e3f3d7d7f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27d668791c44620931035e3f3d7d7f0" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 44583 } } }
I20250109 21:03:02.266670 14573 sys_catalog.cc:458] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:02.266793 14580 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:03:02.269771 14580 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:03:02.275254 14580 catalog_manager.cc:1348] Generated new cluster ID: 7a9879c816fb4ce19b856956a0038a6e
I20250109 21:03:02.275319 14580 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:03:02.288206 14580 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:03:02.289098 14580 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:03:02.297662 14580 catalog_manager.cc:5899] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0: Generated new TSK 0
I20250109 21:03:02.298326 14580 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
I20250109 21:03:02.324280 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:44583
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:02.432736 14592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:02.433017 14592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:02.433101 14592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:02.436470 14592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:02.436550 14592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:02.436643 14592 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20250109 21:03:02.436672 14592 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250109 21:03:02.436738 14592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:03:02.441054 14592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.126.126:44583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:02.442138 14592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:02.443048 14592 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:02.451526 14597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:02.451658 14600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:02.452410 14598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:02.453250 14592 server_base.cc:1034] running on GCE node
I20250109 21:03:02.453672 14592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:02.454255 14592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:02.455397 14592 hybrid_clock.cc:648] HybridClock initialized: now 1736456582455387 us; error 27 us; skew 500 ppm
Jan 09 21:03:02 dist-test-slave-g0zb krb5kdc[14475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456582, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:02.458053 14592 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:03:02.459015 14592 webserver.cc:458] Webserver started at http://127.12.126.65:46551/ using document root <none> and password file <none>
I20250109 21:03:02.459553 14592 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:02.459604 14592 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:02.459746 14592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:02.461493 14592 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "982b8e3e37334d01828202cea3efaba1"
format_stamp: "Formatted at 2025-01-09 21:03:02 on dist-test-slave-g0zb"
server_key: "963445d2af8e13cd82094e703e90f2e6"
server_key_iv: "6a978f0fcf946211ae4ef4045bf51330"
server_key_version: "encryptionkey@0"
I20250109 21:03:02.461934 14592 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "982b8e3e37334d01828202cea3efaba1"
format_stamp: "Formatted at 2025-01-09 21:03:02 on dist-test-slave-g0zb"
server_key: "963445d2af8e13cd82094e703e90f2e6"
server_key_iv: "6a978f0fcf946211ae4ef4045bf51330"
server_key_version: "encryptionkey@0"
I20250109 21:03:02.465368 14592 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250109 21:03:02.467557 14607 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:02.468526 14592 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:02.468617 14592 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "982b8e3e37334d01828202cea3efaba1"
format_stamp: "Formatted at 2025-01-09 21:03:02 on dist-test-slave-g0zb"
server_key: "963445d2af8e13cd82094e703e90f2e6"
server_key_iv: "6a978f0fcf946211ae4ef4045bf51330"
server_key_version: "encryptionkey@0"
I20250109 21:03:02.468695 14592 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:02.481339 14592 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:02.484325 14592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:02.484462 14592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:02.485714 14592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:02.485765 14592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:02.485805 14592 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:02.485852 14592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 09 21:03:02 dist-test-slave-g0zb krb5kdc[14475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456582, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:02.496362 14592 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:42237
I20250109 21:03:02.496373 14722 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:42237 every 8 connection(s)
I20250109 21:03:02.497495 14592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:03:02.498783 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.486327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:60935 (local address 127.12.126.126:44583)
0109 21:03:02.486485 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:02.486489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:02.487403 (+ 914us) server_negotiation.cc:407] Connection header received
0109 21:03:02.488287 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:02.488291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:02.488353 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:02.488448 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:02.490218 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.490920 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.491687 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.491867 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.494821 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:02.494844 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:02.494847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:02.494878 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:02.496712 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:02.497184 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:02.497190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:02.497194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:02.497251 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:02.497635 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:02.497639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:02.497641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:02.497848 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:02.497955 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:02.498526 (+ 571us) server_negotiation.cc:299] Negotiation successful
0109 21:03:02.498646 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250109 21:03:02.499346 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.486739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:44583 (local address 127.12.126.65:60935)
0109 21:03:02.487259 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:02.487302 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:02.488077 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:02.488678 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:02.488691 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:02.489210 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:02.490027 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:02.490048 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.491039 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.491045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:02.491566 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:02.491575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.491787 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.492460 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:02.492493 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:02.494599 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:02.496837 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:02.496842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:02.496850 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:02.497080 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:02.497436 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:02.497439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:02.497440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:02.497524 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:02.498009 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:02.498014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:02.498299 (+ 285us) client_negotiation.cc:769] Sending connection context
0109 21:03:02.498464 (+ 165us) client_negotiation.cc:240] Negotiation successful
0109 21:03:02.498680 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":122,"threads_started":1}
I20250109 21:03:02.500469 14723 heartbeater.cc:346] Connected to a master server at 127.12.126.126:44583
I20250109 21:03:02.500598 14723 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:02.501063 14723 heartbeater.cc:510] Master 127.12.126.126:44583 requested a full tablet report, sending...
I20250109 21:03:02.502518 14519 ts_manager.cc:194] Registered new tserver with Master: 982b8e3e37334d01828202cea3efaba1 (127.12.126.65:42237)
I20250109 21:03:02.503952 14519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:60935
I20250109 21:03:02.504671 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 14592
I20250109 21:03:02.504789 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:03:02.505069 12793 external_mini_cluster.cc:1453] Setting key bc1e6ff885a439e7a823645a14bad8cc
I20250109 21:03:02.509187 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.502427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:40877 (local address 127.12.126.126:44583)
0109 21:03:02.502911 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:02.502915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:02.502925 (+ 10us) server_negotiation.cc:407] Connection header received
0109 21:03:02.502952 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:02.502954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:02.502990 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:02.503052 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:02.503695 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.504226 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.505009 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.505190 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.506048 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:02.506063 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:02.506066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:02.506090 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:02.507884 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:02.508248 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:02.508253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:02.508256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:02.508315 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:02.508505 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:02.508507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:02.508509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:02.508648 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:02.508724 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:02.508992 (+ 268us) server_negotiation.cc:299] Negotiation successful
0109 21:03:02.509079 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408}
I20250109 21:03:02.509377 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.502370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:44583 (local address 127.12.126.65:40877)
0109 21:03:02.502633 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:02.502651 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:02.502730 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:02.503094 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:02.503097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:02.503288 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:02.503591 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:02.503598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.504344 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.504348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:02.504892 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:02.504902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.505019 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.505579 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:02.505594 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:02.505934 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:02.507993 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:02.507996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:02.507997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:02.508161 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:02.508392 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:02.508394 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:02.508395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:02.508431 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:02.508735 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:02.508741 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:02.508836 (+ 95us) client_negotiation.cc:769] Sending connection context
0109 21:03:02.508948 (+ 112us) client_negotiation.cc:240] Negotiation successful
0109 21:03:02.509089 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.12.126.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
I20250109 21:03:02.529347 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:44583
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:02.633981 14732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:02.634199 14732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:02.634259 14732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:02.637303 14732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:02.637368 14732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:02.637418 14732 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20250109 21:03:02.637444 14732 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250109 21:03:02.637495 14732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:03:02.641571 14732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.126.126:44583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:02.642565 14732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:02.643361 14732 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:02.651051 14738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:02.651727 14740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:02.652539 14737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:02.653255 14732 server_base.cc:1034] running on GCE node
I20250109 21:03:02.653645 14732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:02.654214 14732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:02.655361 14732 hybrid_clock.cc:648] HybridClock initialized: now 1736456582655349 us; error 27 us; skew 500 ppm
Jan 09 21:03:02 dist-test-slave-g0zb krb5kdc[14475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456582, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:02.657907 14732 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:03:02.658783 14732 webserver.cc:458] Webserver started at http://127.12.126.66:41091/ using document root <none> and password file <none>
I20250109 21:03:02.659354 14732 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:02.659415 14732 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:02.659595 14732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:02.661335 14732 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "fee46231170a4a3c918e358cff7d3ff5"
format_stamp: "Formatted at 2025-01-09 21:03:02 on dist-test-slave-g0zb"
server_key: "3a63bfd4cb48606d327aa0f621daf7cf"
server_key_iv: "a6f5d38fd21a69813bd5d00c7ab4bb3f"
server_key_version: "encryptionkey@0"
I20250109 21:03:02.661803 14732 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "fee46231170a4a3c918e358cff7d3ff5"
format_stamp: "Formatted at 2025-01-09 21:03:02 on dist-test-slave-g0zb"
server_key: "3a63bfd4cb48606d327aa0f621daf7cf"
server_key_iv: "a6f5d38fd21a69813bd5d00c7ab4bb3f"
server_key_version: "encryptionkey@0"
I20250109 21:03:02.665087 14732 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20250109 21:03:02.667249 14747 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:02.668169 14732 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250109 21:03:02.668272 14732 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "fee46231170a4a3c918e358cff7d3ff5"
format_stamp: "Formatted at 2025-01-09 21:03:02 on dist-test-slave-g0zb"
server_key: "3a63bfd4cb48606d327aa0f621daf7cf"
server_key_iv: "a6f5d38fd21a69813bd5d00c7ab4bb3f"
server_key_version: "encryptionkey@0"
I20250109 21:03:02.668362 14732 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:02.704890 14732 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:02.707554 14732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:02.707693 14732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:02.708951 14732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:02.709003 14732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:02.709065 14732 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:02.709098 14732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 09 21:03:02 dist-test-slave-g0zb krb5kdc[14475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456582, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:02.719632 14732 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:46033
I20250109 21:03:02.719688 14862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:46033 every 8 connection(s)
I20250109 21:03:02.720805 14732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:03:02.722632 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.709745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:38989 (local address 127.12.126.126:44583)
0109 21:03:02.709861 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:02.709865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:02.710733 (+ 868us) server_negotiation.cc:407] Connection header received
0109 21:03:02.711597 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:02.711604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:02.711679 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:02.711793 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:02.713659 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.714425 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.715370 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.715580 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.719009 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:02.719031 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:02.719035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:02.719063 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:02.720736 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:02.721224 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:02.721227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:02.721228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:02.721290 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:02.721637 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:02.721640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:02.721642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:02.721801 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:02.721882 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:02.722406 (+ 524us) server_negotiation.cc:299] Negotiation successful
0109 21:03:02.722512 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250109 21:03:02.723222 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.710145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:44583 (local address 127.12.126.66:38989)
0109 21:03:02.710588 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:02.710621 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:02.711372 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:02.712128 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:02.712142 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:02.712711 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:02.713471 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:02.713505 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.714545 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.714551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:02.715244 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:02.715257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.715522 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.716148 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:02.716185 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:02.718802 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:02.720847 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:02.720855 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:02.720867 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:02.721114 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:02.721372 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:02.721375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:02.721377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:02.721490 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:02.721893 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:02.721897 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:02.722184 (+ 287us) client_negotiation.cc:769] Sending connection context
0109 21:03:02.722353 (+ 169us) client_negotiation.cc:240] Negotiation successful
0109 21:03:02.722575 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":135,"threads_started":1}
I20250109 21:03:02.724208 14863 heartbeater.cc:346] Connected to a master server at 127.12.126.126:44583
I20250109 21:03:02.724334 14863 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:02.724829 14863 heartbeater.cc:510] Master 127.12.126.126:44583 requested a full tablet report, sending...
I20250109 21:03:02.725080 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 14732
I20250109 21:03:02.725189 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:03:02.725477 12793 external_mini_cluster.cc:1453] Setting key 104995fee1624a4718508adc0bf0dde5
I20250109 21:03:02.725782 14520 ts_manager.cc:194] Registered new tserver with Master: fee46231170a4a3c918e358cff7d3ff5 (127.12.126.66:46033)
I20250109 21:03:02.726523 14520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:38989
I20250109 21:03:02.732831 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.726518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:41591 (local address 127.12.126.126:44583)
0109 21:03:02.726648 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:02.726654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:02.726669 (+ 15us) server_negotiation.cc:407] Connection header received
0109 21:03:02.726837 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:02.726841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:02.726878 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:02.726956 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:02.727596 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.728335 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.728967 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.729177 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.729985 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:02.729999 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:02.730001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:02.730021 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:02.731494 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:02.731858 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:02.731862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:02.731863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:02.731904 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:02.732108 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:02.732111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:02.732113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:02.732268 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:02.732358 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:02.732588 (+ 230us) server_negotiation.cc:299] Negotiation successful
0109 21:03:02.732695 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250109 21:03:02.732849 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.726275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:44583 (local address 127.12.126.66:41591)
0109 21:03:02.726538 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:02.726554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:02.726707 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:02.726978 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:02.726980 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:02.727188 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:02.727465 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:02.727474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.728475 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.728478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:02.728854 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:02.728862 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.728972 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.729568 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:02.729580 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:02.729850 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:02.731598 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:02.731602 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:02.731604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:02.731769 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:02.731982 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:02.731987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:02.731989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:02.732031 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:02.732359 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:02.732363 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:02.732438 (+ 75us) client_negotiation.cc:769] Sending connection context
0109 21:03:02.732530 (+ 92us) client_negotiation.cc:240] Negotiation successful
0109 21:03:02.732656 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.12.126.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
I20250109 21:03:02.751965 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:44583
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:02.857834 14872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:02.858037 14872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:02.858094 14872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:02.861127 14872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:02.861188 14872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:02.861245 14872 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20250109 21:03:02.861267 14872 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250109 21:03:02.861306 14872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:03:02.865568 14872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.126.126:44583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:02.866690 14872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:02.867622 14872 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:02.875389 14880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:02.876274 14877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:02.876886 14878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:02.877861 14872 server_base.cc:1034] running on GCE node
I20250109 21:03:02.878211 14872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:02.878718 14872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:02.879865 14872 hybrid_clock.cc:648] HybridClock initialized: now 1736456582879853 us; error 28 us; skew 500 ppm
Jan 09 21:03:02 dist-test-slave-g0zb krb5kdc[14475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456582, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:02.882524 14872 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:03:02.883412 14872 webserver.cc:458] Webserver started at http://127.12.126.67:46871/ using document root <none> and password file <none>
I20250109 21:03:02.883913 14872 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:02.883962 14872 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:02.884111 14872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:02.885834 14872 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "221f5fb33e21499cad02660e45d23242"
format_stamp: "Formatted at 2025-01-09 21:03:02 on dist-test-slave-g0zb"
server_key: "9d56b742d406e70169fb52c650d4eba2"
server_key_iv: "85047a9f43e049c876da6bbdf5493e77"
server_key_version: "encryptionkey@0"
I20250109 21:03:02.886271 14872 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "221f5fb33e21499cad02660e45d23242"
format_stamp: "Formatted at 2025-01-09 21:03:02 on dist-test-slave-g0zb"
server_key: "9d56b742d406e70169fb52c650d4eba2"
server_key_iv: "85047a9f43e049c876da6bbdf5493e77"
server_key_version: "encryptionkey@0"
I20250109 21:03:02.889533 14872 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250109 21:03:02.891577 14887 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:02.892597 14872 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:02.892694 14872 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "221f5fb33e21499cad02660e45d23242"
format_stamp: "Formatted at 2025-01-09 21:03:02 on dist-test-slave-g0zb"
server_key: "9d56b742d406e70169fb52c650d4eba2"
server_key_iv: "85047a9f43e049c876da6bbdf5493e77"
server_key_version: "encryptionkey@0"
I20250109 21:03:02.892781 14872 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:02.907503 14872 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:02.910426 14872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:02.910573 14872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:02.911900 14872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:02.911958 14872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:02.912019 14872 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:02.912045 14872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 09 21:03:02 dist-test-slave-g0zb krb5kdc[14475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456582, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:02.922363 14872 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:40451
I20250109 21:03:02.922976 15002 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:40451 every 8 connection(s)
I20250109 21:03:02.923473 14872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:03:02.924805 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.912475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:48223 (local address 127.12.126.126:44583)
0109 21:03:02.912630 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:02.912635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:02.913338 (+ 703us) server_negotiation.cc:407] Connection header received
0109 21:03:02.914194 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:02.914197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:02.914263 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:02.914368 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:02.915986 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.916831 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.917651 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.917838 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.921026 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:02.921050 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:02.921056 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:02.921096 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:02.922698 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:02.923346 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:02.923349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:02.923351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:02.923400 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:02.923708 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:02.923710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:02.923711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:02.923865 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:02.923956 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:02.924576 (+ 620us) server_negotiation.cc:299] Negotiation successful
0109 21:03:02.924683 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20250109 21:03:02.925441 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.912758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:44583 (local address 127.12.126.67:48223)
0109 21:03:02.913177 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:02.913217 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:02.913987 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:02.914705 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:02.914718 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:02.915198 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:02.915827 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:02.915845 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.916964 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.916970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:02.917538 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:02.917550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.917758 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.918431 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:02.918466 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:02.920865 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:02.922826 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:02.922831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:02.922842 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:02.923246 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:02.923493 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:02.923496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:02.923499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:02.923616 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:02.924010 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:02.924016 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:02.924289 (+ 273us) client_negotiation.cc:769] Sending connection context
0109 21:03:02.924492 (+ 203us) client_negotiation.cc:240] Negotiation successful
0109 21:03:02.924714 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
I20250109 21:03:02.926584 15003 heartbeater.cc:346] Connected to a master server at 127.12.126.126:44583
I20250109 21:03:02.926743 15003 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:02.927371 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 14872
I20250109 21:03:02.927408 15003 heartbeater.cc:510] Master 127.12.126.126:44583 requested a full tablet report, sending...
I20250109 21:03:02.927481 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:03:02.927744 12793 external_mini_cluster.cc:1453] Setting key b77c9d68fe2ccd2b43d178ec7afec188
I20250109 21:03:02.928714 14520 ts_manager.cc:194] Registered new tserver with Master: 221f5fb33e21499cad02660e45d23242 (127.12.126.67:40451)
I20250109 21:03:02.929556 14520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:48223
I20250109 21:03:02.930285 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 21:03:02.938408 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.929546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:44583 (local address 127.12.126.67:36695)
0109 21:03:02.929884 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:02.929899 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:02.930236 (+ 337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:02.931321 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:02.931323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:02.931534 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:02.931804 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:02.931813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.932712 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.932717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:02.933323 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:02.933336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.933798 (+ 462us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.934434 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:02.934453 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:02.934890 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:02.937177 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:02.937181 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:02.937182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:02.937356 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:02.937610 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:02.937615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:02.937618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:02.937662 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:02.937978 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:02.937981 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:02.938058 (+ 77us) client_negotiation.cc:769] Sending connection context
0109 21:03:02.938146 (+ 88us) client_negotiation.cc:240] Negotiation successful
0109 21:03:02.938263 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":96,"threads_started":1}
I20250109 21:03:02.938607 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.929186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:36695 (local address 127.12.126.126:44583)
0109 21:03:02.929319 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:02.929325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:02.929963 (+ 638us) server_negotiation.cc:407] Connection header received
0109 21:03:02.930372 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:02.930377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:02.930423 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:02.930476 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:02.931910 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.932548 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.933462 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.933672 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.935031 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:02.935050 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:02.935053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:02.935079 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:02.937023 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:02.937476 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:02.937480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:02.937481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:02.937526 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:02.937740 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:02.937743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:02.937745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:02.937893 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:02.938002 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:02.938225 (+ 223us) server_negotiation.cc:299] Negotiation successful
0109 21:03:02.938356 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Jan 09 21:03:02 dist-test-slave-g0zb krb5kdc[14475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 09 21:03:02 dist-test-slave-g0zb krb5kdc[14475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:02.956157 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.948770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49284 (local address 127.12.126.126:44583)
0109 21:03:02.948896 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:02.948899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:02.948912 (+ 13us) server_negotiation.cc:407] Connection header received
0109 21:03:02.948950 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:02.948952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:02.948994 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:02.949065 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:02.949838 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.950354 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.951009 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.951156 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.953344 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:02.953361 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:02.953364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:02.953389 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:02.954703 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:02.955117 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:02.955123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:02.955125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:02.955167 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:02.955377 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:02.955380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:02.955383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:02.955531 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:02.955657 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:02.955879 (+ 222us) server_negotiation.cc:299] Negotiation successful
0109 21:03:02.955992 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250109 21:03:02.959069 14519 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:03:02.961666 14519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:03:02.973687 15020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.969452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:40451 (local address 127.0.0.1:59910)
0109 21:03:02.969818 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:02.969860 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:02.969991 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:02.970654 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:02.970660 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:02.970687 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:02.971004 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:02.971018 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.972194 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.972197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:02.973202 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:02.973213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.973312 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.973368 (+ 56us) client_negotiation.cc:769] Sending connection context
0109 21:03:02.973462 (+ 94us) client_negotiation.cc:240] Negotiation successful
0109 21:03:02.973544 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":107,"threads_started":1}
I20250109 21:03:02.974509 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.970162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:46033 (local address 127.0.0.1:44552)
0109 21:03:02.970972 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:02.970984 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:02.971076 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:02.971474 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:02.971477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:02.971502 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:02.971731 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:02.971741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.973298 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.973301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:02.974139 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:02.974149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.974259 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.974276 (+ 17us) client_negotiation.cc:769] Sending connection context
0109 21:03:02.974326 (+ 50us) client_negotiation.cc:240] Negotiation successful
0109 21:03:02.974390 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":111,"threads_started":1}
I20250109 21:03:02.974787 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.970511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:42237 (local address 127.0.0.1:40588)
0109 21:03:02.971258 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:02.971273 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:02.971353 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:02.972107 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:02.972109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:02.972127 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:02.972345 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:02.972354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.973603 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.973606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:02.974428 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:02.974438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.974544 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.974562 (+ 18us) client_negotiation.cc:769] Sending connection context
0109 21:03:02.974613 (+ 51us) client_negotiation.cc:240] Negotiation successful
0109 21:03:02.974665 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":85,"threads_started":1}
I20250109 21:03:02.974794 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.969615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59910 (local address 127.12.126.67:40451)
0109 21:03:02.969944 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:02.969950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:02.969976 (+ 26us) server_negotiation.cc:407] Connection header received
0109 21:03:02.970174 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:02.970179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:02.970377 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:02.970548 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:02.971149 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.972043 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.973879 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.974443 (+ 564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.974559 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:02.974629 (+ 70us) server_negotiation.cc:299] Negotiation successful
0109 21:03:02.974704 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":107,"threads_started":1}
I20250109 21:03:02.975212 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.970372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44552 (local address 127.12.126.66:46033)
0109 21:03:02.970925 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:02.970931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:02.971056 (+ 125us) server_negotiation.cc:407] Connection header received
0109 21:03:02.971212 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:02.971216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:02.971350 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:02.971481 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:02.971978 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.973123 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.974274 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.974837 (+ 563us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.974947 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:02.975013 (+ 66us) server_negotiation.cc:299] Negotiation successful
0109 21:03:02.975091 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":93,"threads_started":1}
I20250109 21:03:02.975854 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:02.970942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40588 (local address 127.12.126.65:42237)
0109 21:03:02.971717 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:02.971722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:02.971742 (+ 20us) server_negotiation.cc:407] Connection header received
0109 21:03:02.971828 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:02.971832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:02.971990 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:02.972113 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:02.972497 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.973492 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:02.974965 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:02.975497 (+ 532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:02.975595 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:02.975657 (+ 62us) server_negotiation.cc:299] Negotiation successful
0109 21:03:02.975730 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":70,"threads_started":1}
I20250109 21:03:02.977346 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 020c05560d674942af9a6453bbf084b6 (DEFAULT_TABLE table=test-table [id=ad7f0969ca144397a3445cd7edae9715]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:02.978179 14937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 020c05560d674942af9a6453bbf084b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:02.978029 14657 tablet_service.cc:1467] Processing CreateTablet for tablet 020c05560d674942af9a6453bbf084b6 (DEFAULT_TABLE table=test-table [id=ad7f0969ca144397a3445cd7edae9715]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:02.978348 14797 tablet_service.cc:1467] Processing CreateTablet for tablet 020c05560d674942af9a6453bbf084b6 (DEFAULT_TABLE table=test-table [id=ad7f0969ca144397a3445cd7edae9715]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:02.978972 14657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 020c05560d674942af9a6453bbf084b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:02.979175 14797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 020c05560d674942af9a6453bbf084b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:02.983672 15026 tablet_bootstrap.cc:492] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242: Bootstrap starting.
I20250109 21:03:02.985491 15027 tablet_bootstrap.cc:492] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5: Bootstrap starting.
I20250109 21:03:02.986876 15026 tablet_bootstrap.cc:654] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:02.987114 15028 tablet_bootstrap.cc:492] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1: Bootstrap starting.
I20250109 21:03:02.987509 15027 tablet_bootstrap.cc:654] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:02.987717 15026 log.cc:826] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:02.988258 15027 log.cc:826] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:02.989363 15028 tablet_bootstrap.cc:654] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:02.989523 15026 tablet_bootstrap.cc:492] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242: No bootstrap required, opened a new log
I20250109 21:03:02.989693 15026 ts_tablet_manager.cc:1397] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250109 21:03:02.990015 15028 log.cc:826] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:02.992161 15028 tablet_bootstrap.cc:492] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1: No bootstrap required, opened a new log
I20250109 21:03:02.992285 15027 tablet_bootstrap.cc:492] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5: No bootstrap required, opened a new log
I20250109 21:03:02.992326 15028 ts_tablet_manager.cc:1397] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:03:02.992480 15027 ts_tablet_manager.cc:1397] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20250109 21:03:02.992470 15026 raft_consensus.cc:357] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } }
I20250109 21:03:02.992691 15026 raft_consensus.cc:383] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:02.992756 15026 raft_consensus.cc:738] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221f5fb33e21499cad02660e45d23242, State: Initialized, Role: FOLLOWER
I20250109 21:03:02.993188 15026 consensus_queue.cc:260] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } }
I20250109 21:03:02.994040 15026 ts_tablet_manager.cc:1428] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250109 21:03:02.994446 15003 heartbeater.cc:502] Master 127.12.126.126:44583 was elected leader, sending a full tablet report...
I20250109 21:03:02.995249 15028 raft_consensus.cc:357] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } }
I20250109 21:03:02.995456 15028 raft_consensus.cc:383] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:02.995343 15027 raft_consensus.cc:357] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } }
I20250109 21:03:02.995515 15028 raft_consensus.cc:738] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982b8e3e37334d01828202cea3efaba1, State: Initialized, Role: FOLLOWER
I20250109 21:03:02.995545 15027 raft_consensus.cc:383] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:02.995610 15027 raft_consensus.cc:738] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee46231170a4a3c918e358cff7d3ff5, State: Initialized, Role: FOLLOWER
I20250109 21:03:02.995980 15028 consensus_queue.cc:260] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } }
I20250109 21:03:02.996004 15027 consensus_queue.cc:260] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } }
I20250109 21:03:02.996726 15028 ts_tablet_manager.cc:1428] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250109 21:03:02.996768 15027 ts_tablet_manager.cc:1428] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250109 21:03:02.997185 14863 heartbeater.cc:502] Master 127.12.126.126:44583 was elected leader, sending a full tablet report...
I20250109 21:03:02.997483 14723 heartbeater.cc:502] Master 127.12.126.126:44583 was elected leader, sending a full tablet report...
W20250109 21:03:02.998517 14724 tablet.cc:2367] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250109 21:03:03.174424 15004 tablet.cc:2367] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250109 21:03:03.222155 14864 tablet.cc:2367] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:03:03.283740 15034 raft_consensus.cc:491] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:03:03.283951 15034 raft_consensus.cc:513] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } }
I20250109 21:03:03.285005 15034 leader_election.cc:290] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 221f5fb33e21499cad02660e45d23242 (127.12.126.67:40451), 982b8e3e37334d01828202cea3efaba1 (127.12.126.65:42237)
I20250109 21:03:03.288189 15035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.285281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:40451 (local address 127.12.126.66:56783)
0109 21:03:03.285569 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:03.285587 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:03.285726 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:03.286072 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:03.286074 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:03.286087 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:03.286300 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.286305 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.287145 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.287151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:03.287760 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.287769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.287864 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.287887 (+ 23us) client_negotiation.cc:769] Sending connection context
0109 21:03:03.287940 (+ 53us) client_negotiation.cc:240] Negotiation successful
0109 21:03:03.287999 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":94,"threads_started":1}
I20250109 21:03:03.288223 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.285635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:42237 (local address 127.12.126.66:42827)
0109 21:03:03.285855 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:03.285868 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:03.285963 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:03.286204 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:03.286207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:03.286224 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:03.286470 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.286492 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.287238 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.287241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:03.287808 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.287816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.287929 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.287950 (+ 21us) client_negotiation.cc:769] Sending connection context
0109 21:03:03.288002 (+ 52us) client_negotiation.cc:240] Negotiation successful
0109 21:03:03.288057 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":55,"threads_started":1}
I20250109 21:03:03.288635 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.285762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:42827 (local address 127.12.126.65:42237)
0109 21:03:03.285909 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:03.285915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:03.285930 (+ 15us) server_negotiation.cc:407] Connection header received
0109 21:03:03.286077 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:03.286079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:03.286120 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:03.286186 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:03.286612 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.287116 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.287933 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.288406 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.288444 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:03.288494 (+ 50us) server_negotiation.cc:299] Negotiation successful
0109 21:03:03.288550 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250109 21:03:03.288738 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.285382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:56783 (local address 127.12.126.67:40451)
0109 21:03:03.285547 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:03.285552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:03.285687 (+ 135us) server_negotiation.cc:407] Connection header received
0109 21:03:03.285869 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:03.285874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:03.285918 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:03.285959 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:03.286412 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.287027 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.287888 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.288428 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.288469 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:03.288524 (+ 55us) server_negotiation.cc:299] Negotiation successful
0109 21:03:03.288592 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20250109 21:03:03.289218 14677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020c05560d674942af9a6453bbf084b6" candidate_uuid: "fee46231170a4a3c918e358cff7d3ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982b8e3e37334d01828202cea3efaba1" is_pre_election: true
I20250109 21:03:03.289295 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020c05560d674942af9a6453bbf084b6" candidate_uuid: "fee46231170a4a3c918e358cff7d3ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "221f5fb33e21499cad02660e45d23242" is_pre_election: true
I20250109 21:03:03.289510 14677 raft_consensus.cc:2463] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fee46231170a4a3c918e358cff7d3ff5 in term 0.
I20250109 21:03:03.289552 14957 raft_consensus.cc:2463] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fee46231170a4a3c918e358cff7d3ff5 in term 0.
I20250109 21:03:03.289953 14749 leader_election.cc:304] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 221f5fb33e21499cad02660e45d23242, fee46231170a4a3c918e358cff7d3ff5; no voters:
I20250109 21:03:03.290202 15034 raft_consensus.cc:2798] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:03:03.290272 15034 raft_consensus.cc:491] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:03:03.290303 15034 raft_consensus.cc:3054] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:03.291314 15034 raft_consensus.cc:513] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } }
I20250109 21:03:03.291656 15034 leader_election.cc:290] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 221f5fb33e21499cad02660e45d23242 (127.12.126.67:40451), 982b8e3e37334d01828202cea3efaba1 (127.12.126.65:42237)
I20250109 21:03:03.292001 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020c05560d674942af9a6453bbf084b6" candidate_uuid: "fee46231170a4a3c918e358cff7d3ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "221f5fb33e21499cad02660e45d23242"
I20250109 21:03:03.292098 14677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020c05560d674942af9a6453bbf084b6" candidate_uuid: "fee46231170a4a3c918e358cff7d3ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982b8e3e37334d01828202cea3efaba1"
I20250109 21:03:03.292117 14957 raft_consensus.cc:3054] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:03.292220 14677 raft_consensus.cc:3054] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:03.293084 14957 raft_consensus.cc:2463] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fee46231170a4a3c918e358cff7d3ff5 in term 1.
I20250109 21:03:03.293157 14677 raft_consensus.cc:2463] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fee46231170a4a3c918e358cff7d3ff5 in term 1.
I20250109 21:03:03.293423 14749 leader_election.cc:304] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 221f5fb33e21499cad02660e45d23242, fee46231170a4a3c918e358cff7d3ff5; no voters:
I20250109 21:03:03.293618 15034 raft_consensus.cc:2798] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:03.293813 15034 raft_consensus.cc:695] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [term 1 LEADER]: Becoming Leader. State: Replica: fee46231170a4a3c918e358cff7d3ff5, State: Running, Role: LEADER
I20250109 21:03:03.294134 15034 consensus_queue.cc:237] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } }
I20250109 21:03:03.297140 14519 catalog_manager.cc:5526] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fee46231170a4a3c918e358cff7d3ff5 (127.12.126.66). New cstate: current_term: 1 leader_uuid: "fee46231170a4a3c918e358cff7d3ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } health_report { overall_health: UNKNOWN } } }
Jan 09 21:03:03 dist-test-slave-g0zb krb5kdc[14475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456582, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:03.355975 15044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.348582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:44583 (local address 127.0.0.1:49288)
0109 21:03:03.348855 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:03.348871 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:03.348976 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:03.349210 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:03.349212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:03.349651 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:03.349848 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.349855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.350525 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.350527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:03.350908 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.350915 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.350998 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.351586 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:03.351600 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:03.353052 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:03.354698 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:03.354703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:03.354708 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:03.354977 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:03.355154 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:03.355155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:03.355157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:03.355199 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:03.355480 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:03.355485 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:03.355576 (+ 91us) client_negotiation.cc:769] Sending connection context
0109 21:03:03.355660 (+ 84us) client_negotiation.cc:240] Negotiation successful
0109 21:03:03.355812 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":95,"threads_started":1}
I20250109 21:03:03.355998 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.348656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.12.126.126:44583)
0109 21:03:03.348784 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:03.348788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:03.348935 (+ 147us) server_negotiation.cc:407] Connection header received
0109 21:03:03.349056 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:03.349059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:03.349101 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:03.349182 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:03.349946 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.350432 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.350985 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.351100 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.353180 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:03.353195 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:03.353198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:03.353219 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:03.354612 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:03.355051 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:03.355053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:03.355054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:03.355089 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:03.355252 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:03.355253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:03.355254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:03.355392 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:03.355470 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:03.355690 (+ 220us) server_negotiation.cc:299] Negotiation successful
0109 21:03:03.355870 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":48}
I20250109 21:03:03.359850 14519 catalog_manager.cc:1909] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:49288:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250109 21:03:03.360579 14519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:03:03.365386 14797 tablet_service.cc:1467] Processing CreateTablet for tablet 680e79735b1c4477ae2794769d42761b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ca6daa29ae09472592be17dd9d7adf41]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250109 21:03:03.365403 14657 tablet_service.cc:1467] Processing CreateTablet for tablet 680e79735b1c4477ae2794769d42761b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ca6daa29ae09472592be17dd9d7adf41]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250109 21:03:03.365677 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 680e79735b1c4477ae2794769d42761b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ca6daa29ae09472592be17dd9d7adf41]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250109 21:03:03.366039 14797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 680e79735b1c4477ae2794769d42761b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:03.366047 14657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 680e79735b1c4477ae2794769d42761b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:03.366153 14937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 680e79735b1c4477ae2794769d42761b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:03.368430 15027 tablet_bootstrap.cc:492] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5: Bootstrap starting.
I20250109 21:03:03.368741 15026 tablet_bootstrap.cc:492] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242: Bootstrap starting.
I20250109 21:03:03.369515 15027 tablet_bootstrap.cc:654] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:03.369518 15028 tablet_bootstrap.cc:492] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1: Bootstrap starting.
I20250109 21:03:03.369697 15026 tablet_bootstrap.cc:654] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:03.370530 15028 tablet_bootstrap.cc:654] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:03.370949 15026 tablet_bootstrap.cc:492] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242: No bootstrap required, opened a new log
I20250109 21:03:03.371035 15026 ts_tablet_manager.cc:1397] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250109 21:03:03.371541 15026 raft_consensus.cc:357] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:03.371657 15026 raft_consensus.cc:383] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:03.371694 15026 raft_consensus.cc:738] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221f5fb33e21499cad02660e45d23242, State: Initialized, Role: FOLLOWER
I20250109 21:03:03.371719 15028 tablet_bootstrap.cc:492] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1: No bootstrap required, opened a new log
I20250109 21:03:03.371789 15028 ts_tablet_manager.cc:1397] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250109 21:03:03.371825 15026 consensus_queue.cc:260] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:03.371960 15027 tablet_bootstrap.cc:492] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5: No bootstrap required, opened a new log
I20250109 21:03:03.372036 15027 ts_tablet_manager.cc:1397] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250109 21:03:03.372082 15026 ts_tablet_manager.cc:1428] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250109 21:03:03.372272 15028 raft_consensus.cc:357] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:03.372370 15028 raft_consensus.cc:383] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:03.372411 15028 raft_consensus.cc:738] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982b8e3e37334d01828202cea3efaba1, State: Initialized, Role: FOLLOWER
I20250109 21:03:03.372380 15032 tablet_replica.cc:440] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } }
I20250109 21:03:03.372495 15032 tablet_replica.cc:443] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:03.372541 15028 consensus_queue.cc:260] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:03.372519 15027 raft_consensus.cc:357] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:03.372643 15027 raft_consensus.cc:383] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:03.372680 15027 raft_consensus.cc:738] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee46231170a4a3c918e358cff7d3ff5, State: Initialized, Role: FOLLOWER
I20250109 21:03:03.372808 15028 ts_tablet_manager.cc:1428] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 21:03:03.372831 15027 consensus_queue.cc:260] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:03.373078 15034 tablet_replica.cc:440] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } }
I20250109 21:03:03.373201 15034 tablet_replica.cc:443] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:03.373085 15027 ts_tablet_manager.cc:1428] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 21:03:03.373531 15033 tablet_replica.cc:440] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } }
I20250109 21:03:03.373651 15033 tablet_replica.cc:443] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:03.458508 15033 raft_consensus.cc:491] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:03:03.458664 15033 raft_consensus.cc:513] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:03.459779 15033 leader_election.cc:290] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fee46231170a4a3c918e358cff7d3ff5 (127.12.126.66:46033), 221f5fb33e21499cad02660e45d23242 (127.12.126.67:40451)
I20250109 21:03:03.463671 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.460022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:46033 (local address 127.12.126.65:54791)
0109 21:03:03.460322 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:03.460339 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:03.460457 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:03.460808 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:03.460812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:03.460838 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:03.461132 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.461142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.462184 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.462188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:03.463205 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.463216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.463338 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.463359 (+ 21us) client_negotiation.cc:769] Sending connection context
0109 21:03:03.463413 (+ 54us) client_negotiation.cc:240] Negotiation successful
0109 21:03:03.463487 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"spinlock_wait_cycles":5376,"thread_start_us":106,"threads_started":1}
I20250109 21:03:03.463672 15049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.460438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:40451 (local address 127.12.126.65:33807)
0109 21:03:03.460704 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:03.460714 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:03.460837 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:03.461045 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:03.461047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:03.461062 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:03.461304 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.461310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.462184 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.462188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:03.463204 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.463216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.463337 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.463359 (+ 22us) client_negotiation.cc:769] Sending connection context
0109 21:03:03.463413 (+ 54us) client_negotiation.cc:240] Negotiation successful
0109 21:03:03.463479 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"spinlock_wait_cycles":768,"thread_start_us":73,"threads_started":1}
I20250109 21:03:03.464039 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.460122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:54791 (local address 127.12.126.66:46033)
0109 21:03:03.460298 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:03.460302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:03.460414 (+ 112us) server_negotiation.cc:407] Connection header received
0109 21:03:03.460601 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:03.460605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:03.460659 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:03.460753 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:03.461254 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.462055 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.463321 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.463791 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.463821 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:03.463873 (+ 52us) server_negotiation.cc:299] Negotiation successful
0109 21:03:03.463927 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20250109 21:03:03.464039 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.460542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:33807 (local address 127.12.126.67:40451)
0109 21:03:03.460684 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:03.460688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:03.460814 (+ 126us) server_negotiation.cc:407] Connection header received
0109 21:03:03.460919 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:03.460921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:03.460961 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:03.461022 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:03.461395 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.462075 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.463321 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.463782 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.463821 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:03.463874 (+ 53us) server_negotiation.cc:299] Negotiation successful
0109 21:03:03.463926 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20250109 21:03:03.464505 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680e79735b1c4477ae2794769d42761b" candidate_uuid: "982b8e3e37334d01828202cea3efaba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "221f5fb33e21499cad02660e45d23242" is_pre_election: true
I20250109 21:03:03.464658 14957 raft_consensus.cc:2463] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982b8e3e37334d01828202cea3efaba1 in term 0.
I20250109 21:03:03.464622 14806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680e79735b1c4477ae2794769d42761b" candidate_uuid: "982b8e3e37334d01828202cea3efaba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee46231170a4a3c918e358cff7d3ff5" is_pre_election: true
I20250109 21:03:03.464834 14806 raft_consensus.cc:2463] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982b8e3e37334d01828202cea3efaba1 in term 0.
I20250109 21:03:03.465031 14609 leader_election.cc:304] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 221f5fb33e21499cad02660e45d23242, 982b8e3e37334d01828202cea3efaba1; no voters:
I20250109 21:03:03.465292 15033 raft_consensus.cc:2798] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:03:03.465358 15033 raft_consensus.cc:491] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:03:03.465404 15033 raft_consensus.cc:3054] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:03.466387 15033 raft_consensus.cc:513] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:03.466730 15033 leader_election.cc:290] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [CANDIDATE]: Term 1 election: Requested vote from peers fee46231170a4a3c918e358cff7d3ff5 (127.12.126.66:46033), 221f5fb33e21499cad02660e45d23242 (127.12.126.67:40451)
I20250109 21:03:03.467092 14806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680e79735b1c4477ae2794769d42761b" candidate_uuid: "982b8e3e37334d01828202cea3efaba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee46231170a4a3c918e358cff7d3ff5"
I20250109 21:03:03.467121 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680e79735b1c4477ae2794769d42761b" candidate_uuid: "982b8e3e37334d01828202cea3efaba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "221f5fb33e21499cad02660e45d23242"
I20250109 21:03:03.467222 14806 raft_consensus.cc:3054] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:03.467243 14957 raft_consensus.cc:3054] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:03.468266 14806 raft_consensus.cc:2463] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982b8e3e37334d01828202cea3efaba1 in term 1.
I20250109 21:03:03.468277 14957 raft_consensus.cc:2463] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982b8e3e37334d01828202cea3efaba1 in term 1.
I20250109 21:03:03.468559 14609 leader_election.cc:304] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 982b8e3e37334d01828202cea3efaba1, fee46231170a4a3c918e358cff7d3ff5; no voters:
I20250109 21:03:03.468708 15033 raft_consensus.cc:2798] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:03.468894 15033 raft_consensus.cc:695] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [term 1 LEADER]: Becoming Leader. State: Replica: 982b8e3e37334d01828202cea3efaba1, State: Running, Role: LEADER
I20250109 21:03:03.469195 15033 consensus_queue.cc:237] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:03.471253 15050 tablet_replica.cc:440] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1: TxnStatusTablet state changed. Reason: New leader 982b8e3e37334d01828202cea3efaba1. Latest consensus state: current_term: 1 leader_uuid: "982b8e3e37334d01828202cea3efaba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } }
I20250109 21:03:03.471356 15050 tablet_replica.cc:443] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1: This TxnStatusTablet replica's current role is: LEADER
I20250109 21:03:03.471629 15052 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250109 21:03:03.471827 15052 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250109 21:03:03.472294 14517 catalog_manager.cc:5526] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 982b8e3e37334d01828202cea3efaba1 (127.12.126.65). New cstate: current_term: 1 leader_uuid: "982b8e3e37334d01828202cea3efaba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } health_report { overall_health: UNKNOWN } } }
I20250109 21:03:03.498136 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.494512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.12.126.65:42237)
0109 21:03:03.494634 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:03.494638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:03.494652 (+ 14us) server_negotiation.cc:407] Connection header received
0109 21:03:03.494788 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:03.494791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:03.494895 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:03.495008 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:03.495314 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.496053 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.496943 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.497128 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.497202 (+ 74us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:03.497716 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:03.497844 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:03.497954 (+ 110us) server_negotiation.cc:299] Negotiation successful
0109 21:03:03.498010 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250109 21:03:03.498137 15044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.494377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:42237 (local address 127.0.0.1:40600)
0109 21:03:03.494531 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:03.494549 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:03.494638 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:03.495014 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:03.495016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:03.495023 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0109 21:03:03.495203 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.495208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.496177 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.496180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:03.496769 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.496781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.496895 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.496944 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0109 21:03:03.497838 (+ 894us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0109 21:03:03.497855 (+ 17us) client_negotiation.cc:769] Sending connection context
0109 21:03:03.497934 (+ 79us) client_negotiation.cc:240] Negotiation successful
0109 21:03:03.498011 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20250109 21:03:03.503633 14957 raft_consensus.cc:1270] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242 [term 1 FOLLOWER]: Refusing update from remote peer 982b8e3e37334d01828202cea3efaba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:03.503875 14806 raft_consensus.cc:1270] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5 [term 1 FOLLOWER]: Refusing update from remote peer 982b8e3e37334d01828202cea3efaba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:03.504371 15050 consensus_queue.cc:1035] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:03.504570 15033 consensus_queue.cc:1035] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:03.507899 15034 tablet_replica.cc:440] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5: TxnStatusTablet state changed. Reason: New leader 982b8e3e37334d01828202cea3efaba1. Latest consensus state: current_term: 1 leader_uuid: "982b8e3e37334d01828202cea3efaba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } }
I20250109 21:03:03.508002 15034 tablet_replica.cc:443] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:03.509675 15032 tablet_replica.cc:440] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242: TxnStatusTablet state changed. Reason: New leader 982b8e3e37334d01828202cea3efaba1. Latest consensus state: current_term: 1 leader_uuid: "982b8e3e37334d01828202cea3efaba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } }
I20250109 21:03:03.509791 15032 tablet_replica.cc:443] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:03.512418 15032 tablet_replica.cc:440] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "982b8e3e37334d01828202cea3efaba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } }
I20250109 21:03:03.512387 15033 tablet_replica.cc:440] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "982b8e3e37334d01828202cea3efaba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } }
I20250109 21:03:03.512503 15032 tablet_replica.cc:443] T 680e79735b1c4477ae2794769d42761b P 221f5fb33e21499cad02660e45d23242: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:03.512504 15033 tablet_replica.cc:443] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1: This TxnStatusTablet replica's current role is: LEADER
I20250109 21:03:03.512615 15033 tablet_replica.cc:440] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "982b8e3e37334d01828202cea3efaba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } }
I20250109 21:03:03.512692 15033 tablet_replica.cc:443] T 680e79735b1c4477ae2794769d42761b P 982b8e3e37334d01828202cea3efaba1: This TxnStatusTablet replica's current role is: LEADER
I20250109 21:03:03.513118 15034 tablet_replica.cc:440] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "982b8e3e37334d01828202cea3efaba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } } }
I20250109 21:03:03.513221 15034 tablet_replica.cc:443] T 680e79735b1c4477ae2794769d42761b P fee46231170a4a3c918e358cff7d3ff5: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:03.515666 15057 mvcc.cc:204] Tried to move back new op lower bound from 7112526166024343552 to 7112526165892145152. Current Snapshot: MvccSnapshot[applied={T|T < 7112526166024343552}]
I20250109 21:03:03.516070 15056 mvcc.cc:204] Tried to move back new op lower bound from 7112526166024343552 to 7112526165892145152. Current Snapshot: MvccSnapshot[applied={T|T < 7112526166024343552}]
I20250109 21:03:03.524417 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.521067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.12.126.66:46033)
0109 21:03:03.521203 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:03.521208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:03.521378 (+ 170us) server_negotiation.cc:407] Connection header received
0109 21:03:03.521533 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:03.521536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:03.521586 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:03.521675 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:03.522079 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.522565 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.523466 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.523593 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.523643 (+ 50us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:03.524039 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:03.524123 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:03.524218 (+ 95us) server_negotiation.cc:299] Negotiation successful
0109 21:03:03.524271 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250109 21:03:03.534078 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.531305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:42237 (local address 127.12.126.66:42427)
0109 21:03:03.531586 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:03.531599 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:03.531702 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:03.531932 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:03.531935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:03.531942 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0109 21:03:03.532161 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.532166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.532870 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.532875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:03.533295 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.533303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.533406 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.533448 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0109 21:03:03.533790 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0109 21:03:03.533797 (+ 7us) client_negotiation.cc:769] Sending connection context
0109 21:03:03.533861 (+ 64us) client_negotiation.cc:240] Negotiation successful
0109 21:03:03.533926 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":99,"threads_started":1}
I20250109 21:03:03.534060 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.531407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:42427 (local address 127.12.126.65:42237)
0109 21:03:03.531553 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:03.531558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:03.531661 (+ 103us) server_negotiation.cc:407] Connection header received
0109 21:03:03.531806 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:03.531808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:03.531852 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:03.531917 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:03.532265 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.532744 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.533402 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.533601 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.533645 (+ 44us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:03.533709 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:03.533774 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:03.533876 (+ 102us) server_negotiation.cc:299] Negotiation successful
0109 21:03:03.533920 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250109 21:03:03.540773 14957 raft_consensus.cc:1270] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242 [term 1 FOLLOWER]: Refusing update from remote peer fee46231170a4a3c918e358cff7d3ff5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:03.540879 14677 raft_consensus.cc:1270] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1 [term 1 FOLLOWER]: Refusing update from remote peer fee46231170a4a3c918e358cff7d3ff5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:03.541210 15037 consensus_queue.cc:1035] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:03.541391 15034 consensus_queue.cc:1035] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:03.546301 15055 mvcc.cc:204] Tried to move back new op lower bound from 7112526166178926592 to 7112526165175324672. Current Snapshot: MvccSnapshot[applied={T|T < 7112526166178926592}]
I20250109 21:03:03.564899 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.561869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:42151 (local address 127.12.126.66:46033)
0109 21:03:03.562002 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:03.562007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:03.562022 (+ 15us) server_negotiation.cc:407] Connection header received
0109 21:03:03.562156 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:03.562160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:03.562215 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:03.562290 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:03.562767 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.563462 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.564360 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.564496 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.564534 (+ 38us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:03.564598 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:03.564653 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:03.564754 (+ 101us) server_negotiation.cc:299] Negotiation successful
0109 21:03:03.564793 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250109 21:03:03.564976 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.561593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:46033 (local address 127.12.126.65:42151)
0109 21:03:03.561897 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:03.561914 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:03.562036 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:03.562317 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:03.562320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:03.562330 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0109 21:03:03.562647 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.562655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.563578 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.563582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:03.564258 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.564268 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.564397 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.564444 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0109 21:03:03.564675 (+ 231us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0109 21:03:03.564681 (+ 6us) client_negotiation.cc:769] Sending connection context
0109 21:03:03.564745 (+ 64us) client_negotiation.cc:240] Negotiation successful
0109 21:03:03.564811 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":95,"threads_started":1}
I20250109 21:03:03.591770 14517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49284:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:03.591956 14517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49284:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:03.594488 14517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0: Sending DeleteTablet for 3 replicas of tablet 020c05560d674942af9a6453bbf084b6
I20250109 21:03:03.595456 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet 020c05560d674942af9a6453bbf084b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:03 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:59910
I20250109 21:03:03.595789 14520 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:03:03.596185 14520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:03:03.596326 14797 tablet_service.cc:1514] Processing DeleteTablet for tablet 020c05560d674942af9a6453bbf084b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:03 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:44552
I20250109 21:03:03.596827 14657 tablet_service.cc:1514] Processing DeleteTablet for tablet 020c05560d674942af9a6453bbf084b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:03 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:40588
I20250109 21:03:03.597441 15078 tablet_replica.cc:331] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1: stopping tablet replica
I20250109 21:03:03.597604 15077 tablet_replica.cc:331] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5: stopping tablet replica
I20250109 21:03:03.597896 15078 raft_consensus.cc:2238] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:03.598181 15077 raft_consensus.cc:2238] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:03:03.598582 15077 raft_consensus.cc:2267] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:03.598788 15078 raft_consensus.cc:2267] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:03.600134 15078 ts_tablet_manager.cc:1905] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:03.601500 14797 tablet_service.cc:1467] Processing CreateTablet for tablet a33184ecd1ca4fd387eb7b24b5a5c428 (DEFAULT_TABLE table=test-table [id=e0af624fdafc4931861c287eb35627a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:03.601786 14797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a33184ecd1ca4fd387eb7b24b5a5c428. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:03.602610 15076 tablet_replica.cc:331] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242: stopping tablet replica
I20250109 21:03:03.602890 15076 raft_consensus.cc:2238] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:03.603096 15076 raft_consensus.cc:2267] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:03.603061 14937 tablet_service.cc:1467] Processing CreateTablet for tablet a33184ecd1ca4fd387eb7b24b5a5c428 (DEFAULT_TABLE table=test-table [id=e0af624fdafc4931861c287eb35627a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:03.603317 14937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a33184ecd1ca4fd387eb7b24b5a5c428. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:03.604518 15027 tablet_bootstrap.cc:492] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5: Bootstrap starting.
I20250109 21:03:03.605511 15027 tablet_bootstrap.cc:654] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:03.605549 15026 tablet_bootstrap.cc:492] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242: Bootstrap starting.
I20250109 21:03:03.606458 15026 tablet_bootstrap.cc:654] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:03.607368 15076 ts_tablet_manager.cc:1905] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:03.608265 15077 ts_tablet_manager.cc:1905] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:03.601648 14657 tablet_service.cc:1467] Processing CreateTablet for tablet a33184ecd1ca4fd387eb7b24b5a5c428 (DEFAULT_TABLE table=test-table [id=e0af624fdafc4931861c287eb35627a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:03.608568 14657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a33184ecd1ca4fd387eb7b24b5a5c428. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:03.610054 15076 ts_tablet_manager.cc:1918] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250109 21:03:03.610131 15076 log.cc:1198] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/020c05560d674942af9a6453bbf084b6
I20250109 21:03:03.610436 15076 ts_tablet_manager.cc:1939] T 020c05560d674942af9a6453bbf084b6 P 221f5fb33e21499cad02660e45d23242: Deleting consensus metadata
I20250109 21:03:03.610854 15028 tablet_bootstrap.cc:492] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1: Bootstrap starting.
I20250109 21:03:03.611572 14505 catalog_manager.cc:4872] TS 221f5fb33e21499cad02660e45d23242 (127.12.126.67:40451): tablet 020c05560d674942af9a6453bbf084b6 (table test-table [id=ad7f0969ca144397a3445cd7edae9715]) successfully deleted
I20250109 21:03:03.611965 15028 tablet_bootstrap.cc:654] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:03.612764 15026 tablet_bootstrap.cc:492] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242: No bootstrap required, opened a new log
I20250109 21:03:03.612844 15026 ts_tablet_manager.cc:1397] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20250109 21:03:03.613265 15026 raft_consensus.cc:357] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:03.613372 15026 raft_consensus.cc:383] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:03.613404 15026 raft_consensus.cc:738] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221f5fb33e21499cad02660e45d23242, State: Initialized, Role: FOLLOWER
I20250109 21:03:03.613531 15026 consensus_queue.cc:260] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:03.613780 15026 ts_tablet_manager.cc:1428] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 21:03:03.615154 15027 tablet_bootstrap.cc:492] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5: No bootstrap required, opened a new log
I20250109 21:03:03.615274 15027 ts_tablet_manager.cc:1397] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250109 21:03:03.615770 15027 raft_consensus.cc:357] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:03.615890 15027 raft_consensus.cc:383] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:03.615931 15027 raft_consensus.cc:738] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee46231170a4a3c918e358cff7d3ff5, State: Initialized, Role: FOLLOWER
I20250109 21:03:03.616067 15027 consensus_queue.cc:260] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:03.616343 15027 ts_tablet_manager.cc:1428] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 21:03:03.617383 15028 tablet_bootstrap.cc:492] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1: No bootstrap required, opened a new log
I20250109 21:03:03.617482 15028 ts_tablet_manager.cc:1397] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20250109 21:03:03.617977 15028 raft_consensus.cc:357] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:03.618081 15028 raft_consensus.cc:383] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:03.618115 15028 raft_consensus.cc:738] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982b8e3e37334d01828202cea3efaba1, State: Initialized, Role: FOLLOWER
I20250109 21:03:03.618233 15028 consensus_queue.cc:260] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:03.618373 15078 ts_tablet_manager.cc:1918] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250109 21:03:03.618463 15078 log.cc:1198] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/wal/wals/020c05560d674942af9a6453bbf084b6
I20250109 21:03:03.618460 15077 ts_tablet_manager.cc:1918] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250109 21:03:03.618534 15077 log.cc:1198] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/020c05560d674942af9a6453bbf084b6
I20250109 21:03:03.618532 15028 ts_tablet_manager.cc:1428] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250109 21:03:03.618840 15078 ts_tablet_manager.cc:1939] T 020c05560d674942af9a6453bbf084b6 P 982b8e3e37334d01828202cea3efaba1: Deleting consensus metadata
I20250109 21:03:03.619973 14505 catalog_manager.cc:4872] TS 982b8e3e37334d01828202cea3efaba1 (127.12.126.65:42237): tablet 020c05560d674942af9a6453bbf084b6 (table test-table [id=ad7f0969ca144397a3445cd7edae9715]) successfully deleted
I20250109 21:03:03.620584 15077 ts_tablet_manager.cc:1939] T 020c05560d674942af9a6453bbf084b6 P fee46231170a4a3c918e358cff7d3ff5: Deleting consensus metadata
I20250109 21:03:03.621605 14505 catalog_manager.cc:4872] TS fee46231170a4a3c918e358cff7d3ff5 (127.12.126.66:46033): tablet 020c05560d674942af9a6453bbf084b6 (table test-table [id=ad7f0969ca144397a3445cd7edae9715]) successfully deleted
I20250109 21:03:03.688537 15032 raft_consensus.cc:491] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:03:03.688678 15032 raft_consensus.cc:513] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:03.689756 15032 leader_election.cc:290] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982b8e3e37334d01828202cea3efaba1 (127.12.126.65:42237), fee46231170a4a3c918e358cff7d3ff5 (127.12.126.66:46033)
I20250109 21:03:03.693398 15083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.690370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:46033 (local address 127.12.126.67:35821)
0109 21:03:03.690695 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:03.690707 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:03.690793 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:03.691094 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:03.691096 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:03.691109 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:03.691345 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.691355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.692101 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.692104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:03.692963 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.692972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.693103 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.693132 (+ 29us) client_negotiation.cc:769] Sending connection context
0109 21:03:03.693189 (+ 57us) client_negotiation.cc:240] Negotiation successful
0109 21:03:03.693250 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":77,"threads_started":1}
I20250109 21:03:03.693835 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.690500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:35821 (local address 127.12.126.66:46033)
0109 21:03:03.690648 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:03.690653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:03.690779 (+ 126us) server_negotiation.cc:407] Connection header received
0109 21:03:03.690920 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:03.690923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:03.690986 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:03.691071 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:03.691515 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.691988 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.693089 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.693569 (+ 480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.693609 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:03.693662 (+ 53us) server_negotiation.cc:299] Negotiation successful
0109 21:03:03.693715 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20250109 21:03:03.694106 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.690081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:58563 (local address 127.12.126.65:42237)
0109 21:03:03.690209 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:03.690213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:03.690453 (+ 240us) server_negotiation.cc:407] Connection header received
0109 21:03:03.690684 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:03.690687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:03.690753 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:03.690837 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:03.691441 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.692198 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.693210 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.693691 (+ 481us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.693730 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:03.693910 (+ 180us) server_negotiation.cc:299] Negotiation successful
0109 21:03:03.693994 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250109 21:03:03.694336 15082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.689995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:42237 (local address 127.12.126.67:58563)
0109 21:03:03.690365 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:03.690381 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:03.690537 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:03.690994 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:03.690996 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:03.691010 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:03.691298 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.691306 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.692300 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.692303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:03.693065 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.693072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.693775 (+ 703us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.693790 (+ 15us) client_negotiation.cc:769] Sending connection context
0109 21:03:03.693868 (+ 78us) client_negotiation.cc:240] Negotiation successful
0109 21:03:03.694071 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":121,"threads_started":1}
I20250109 21:03:03.694334 14806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a33184ecd1ca4fd387eb7b24b5a5c428" candidate_uuid: "221f5fb33e21499cad02660e45d23242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee46231170a4a3c918e358cff7d3ff5" is_pre_election: true
I20250109 21:03:03.694476 14806 raft_consensus.cc:2463] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 221f5fb33e21499cad02660e45d23242 in term 0.
I20250109 21:03:03.694870 14889 leader_election.cc:304] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 221f5fb33e21499cad02660e45d23242, fee46231170a4a3c918e358cff7d3ff5; no voters:
I20250109 21:03:03.694983 14677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a33184ecd1ca4fd387eb7b24b5a5c428" candidate_uuid: "221f5fb33e21499cad02660e45d23242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982b8e3e37334d01828202cea3efaba1" is_pre_election: true
I20250109 21:03:03.695106 14677 raft_consensus.cc:2463] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 221f5fb33e21499cad02660e45d23242 in term 0.
I20250109 21:03:03.695113 15032 raft_consensus.cc:2798] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:03:03.695173 15032 raft_consensus.cc:491] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:03:03.695209 15032 raft_consensus.cc:3054] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:03.696346 15032 raft_consensus.cc:513] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:03.696719 15032 leader_election.cc:290] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [CANDIDATE]: Term 1 election: Requested vote from peers 982b8e3e37334d01828202cea3efaba1 (127.12.126.65:42237), fee46231170a4a3c918e358cff7d3ff5 (127.12.126.66:46033)
I20250109 21:03:03.697029 14806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a33184ecd1ca4fd387eb7b24b5a5c428" candidate_uuid: "221f5fb33e21499cad02660e45d23242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee46231170a4a3c918e358cff7d3ff5"
I20250109 21:03:03.697016 14677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a33184ecd1ca4fd387eb7b24b5a5c428" candidate_uuid: "221f5fb33e21499cad02660e45d23242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982b8e3e37334d01828202cea3efaba1"
I20250109 21:03:03.697153 14806 raft_consensus.cc:3054] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:03.697168 14677 raft_consensus.cc:3054] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:03.698168 14677 raft_consensus.cc:2463] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 221f5fb33e21499cad02660e45d23242 in term 1.
I20250109 21:03:03.698168 14806 raft_consensus.cc:2463] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 221f5fb33e21499cad02660e45d23242 in term 1.
I20250109 21:03:03.698464 14889 leader_election.cc:304] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 221f5fb33e21499cad02660e45d23242, 982b8e3e37334d01828202cea3efaba1; no voters:
I20250109 21:03:03.698722 15032 raft_consensus.cc:2798] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:03.698932 15032 raft_consensus.cc:695] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [term 1 LEADER]: Becoming Leader. State: Replica: 221f5fb33e21499cad02660e45d23242, State: Running, Role: LEADER
I20250109 21:03:03.699184 15032 consensus_queue.cc:237] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:03.701467 14517 catalog_manager.cc:5526] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 reported cstate change: term changed from 0 to 1, leader changed from <none> to 221f5fb33e21499cad02660e45d23242 (127.12.126.67). New cstate: current_term: 1 leader_uuid: "221f5fb33e21499cad02660e45d23242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } health_report { overall_health: HEALTHY } } }
I20250109 21:03:03.719718 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.716710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.12.126.67:40451)
0109 21:03:03.716846 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:03.716850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:03.716862 (+ 12us) server_negotiation.cc:407] Connection header received
0109 21:03:03.717007 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:03.717010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:03.717059 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:03.717146 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:03.717468 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.717913 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.718562 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.718738 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.718793 (+ 55us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:03.719218 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:03.719338 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:03.719531 (+ 193us) server_negotiation.cc:299] Negotiation successful
0109 21:03:03.719589 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250109 21:03:03.737259 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.733998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:49751 (local address 127.12.126.65:42237)
0109 21:03:03.734131 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:03.734137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:03.734154 (+ 17us) server_negotiation.cc:407] Connection header received
0109 21:03:03.734216 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:03.734219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:03.734262 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:03.734331 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:03.734853 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.735601 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.736534 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.736730 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.736793 (+ 63us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:03.736885 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:03.736974 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:03.737067 (+ 93us) server_negotiation.cc:299] Negotiation successful
0109 21:03:03.737124 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250109 21:03:03.737273 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.733637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:42237 (local address 127.12.126.67:49751)
0109 21:03:03.733972 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:03.733986 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:03.734087 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:03.734357 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:03.734360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:03.734367 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0109 21:03:03.734694 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.734701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.735721 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.735724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:03.736389 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.736398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.736516 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.736563 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0109 21:03:03.736979 (+ 416us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0109 21:03:03.736984 (+ 5us) client_negotiation.cc:769] Sending connection context
0109 21:03:03.737043 (+ 59us) client_negotiation.cc:240] Negotiation successful
0109 21:03:03.737110 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":139,"threads_started":1}
I20250109 21:03:03.744904 14806 raft_consensus.cc:1270] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5 [term 1 FOLLOWER]: Refusing update from remote peer 221f5fb33e21499cad02660e45d23242: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:03.744990 14677 raft_consensus.cc:1270] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1 [term 1 FOLLOWER]: Refusing update from remote peer 221f5fb33e21499cad02660e45d23242: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:03.745540 15084 consensus_queue.cc:1035] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:03.745766 15032 consensus_queue.cc:1035] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:03.764201 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.761322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:36115 (local address 127.12.126.67:40451)
0109 21:03:03.761429 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:03.761433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:03.761458 (+ 25us) server_negotiation.cc:407] Connection header received
0109 21:03:03.761501 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:03.761504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:03.761549 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:03.761612 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:03.762012 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.762690 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.763585 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.763770 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.763811 (+ 41us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:03.763871 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:03.763938 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:03.764032 (+ 94us) server_negotiation.cc:299] Negotiation successful
0109 21:03:03.764074 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250109 21:03:03.764207 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:03.761059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:40451 (local address 127.12.126.65:36115)
0109 21:03:03.761309 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:03.761325 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:03.761410 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:03.761627 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:03.761630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:03.761637 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0109 21:03:03.761879 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.761888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.762841 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:03.762844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:03.763485 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:03.763493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:03.763597 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:03.763628 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0109 21:03:03.763955 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0109 21:03:03.763962 (+ 7us) client_negotiation.cc:769] Sending connection context
0109 21:03:03.764024 (+ 62us) client_negotiation.cc:240] Negotiation successful
0109 21:03:03.764095 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
W20250109 21:03:03.765170 15056 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20250109 21:03:05.325763 14517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49284:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:05.325918 14517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49284:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:05.328437 14517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0: Sending DeleteTablet for 3 replicas of tablet a33184ecd1ca4fd387eb7b24b5a5c428
I20250109 21:03:05.329061 14657 tablet_service.cc:1514] Processing DeleteTablet for tablet a33184ecd1ca4fd387eb7b24b5a5c428 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:05 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:40588
I20250109 21:03:05.329105 14797 tablet_service.cc:1514] Processing DeleteTablet for tablet a33184ecd1ca4fd387eb7b24b5a5c428 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:05 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:44552
I20250109 21:03:05.329273 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet a33184ecd1ca4fd387eb7b24b5a5c428 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:05 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:59910
I20250109 21:03:05.329613 15114 tablet_replica.cc:331] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1: stopping tablet replica
I20250109 21:03:05.329733 15115 tablet_replica.cc:331] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5: stopping tablet replica
I20250109 21:03:05.329802 15114 raft_consensus.cc:2238] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:05.329901 15115 raft_consensus.cc:2238] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:05.329939 15114 raft_consensus.cc:2267] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:05.330053 15115 raft_consensus.cc:2267] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:05.330651 15114 ts_tablet_manager.cc:1905] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:05.330675 15115 ts_tablet_manager.cc:1905] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:05.333405 15114 ts_tablet_manager.cc:1918] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250109 21:03:05.333500 15114 log.cc:1198] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/wal/wals/a33184ecd1ca4fd387eb7b24b5a5c428
I20250109 21:03:05.333560 15115 ts_tablet_manager.cc:1918] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250109 21:03:05.333644 15115 log.cc:1198] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/a33184ecd1ca4fd387eb7b24b5a5c428
I20250109 21:03:05.333814 15114 ts_tablet_manager.cc:1939] T a33184ecd1ca4fd387eb7b24b5a5c428 P 982b8e3e37334d01828202cea3efaba1: Deleting consensus metadata
I20250109 21:03:05.333936 15115 ts_tablet_manager.cc:1939] T a33184ecd1ca4fd387eb7b24b5a5c428 P fee46231170a4a3c918e358cff7d3ff5: Deleting consensus metadata
I20250109 21:03:05.334844 15116 tablet_replica.cc:331] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242: stopping tablet replica
I20250109 21:03:05.334846 14505 catalog_manager.cc:4872] TS fee46231170a4a3c918e358cff7d3ff5 (127.12.126.66:46033): tablet a33184ecd1ca4fd387eb7b24b5a5c428 (table test-table [id=e0af624fdafc4931861c287eb35627a7]) successfully deleted
I20250109 21:03:05.335091 15116 raft_consensus.cc:2238] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:03:05.335428 14505 catalog_manager.cc:4872] TS 982b8e3e37334d01828202cea3efaba1 (127.12.126.65:42237): tablet a33184ecd1ca4fd387eb7b24b5a5c428 (table test-table [id=e0af624fdafc4931861c287eb35627a7]) successfully deleted
I20250109 21:03:05.335464 15116 raft_consensus.cc:2267] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:05.336167 15116 ts_tablet_manager.cc:1905] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:05.338629 15116 ts_tablet_manager.cc:1918] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250109 21:03:05.338696 15116 log.cc:1198] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/a33184ecd1ca4fd387eb7b24b5a5c428
I20250109 21:03:05.338980 15116 ts_tablet_manager.cc:1939] T a33184ecd1ca4fd387eb7b24b5a5c428 P 221f5fb33e21499cad02660e45d23242: Deleting consensus metadata
I20250109 21:03:05.339689 14505 catalog_manager.cc:4872] TS 221f5fb33e21499cad02660e45d23242 (127.12.126.67:40451): tablet a33184ecd1ca4fd387eb7b24b5a5c428 (table test-table [id=e0af624fdafc4931861c287eb35627a7]) successfully deleted
Jan 09 21:03:05 dist-test-slave-g0zb krb5kdc[14475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456585, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 09 21:03:05 dist-test-slave-g0zb krb5kdc[14475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456585, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:05.366515 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:05.351878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39526 (local address 127.12.126.126:44583)
0109 21:03:05.352180 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:05.352184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:05.352200 (+ 16us) server_negotiation.cc:407] Connection header received
0109 21:03:05.352294 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:05.352297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:05.352345 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:05.352433 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:05.353342 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:05.353860 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:05.354572 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:05.354697 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:05.363378 (+ 8681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:05.363402 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:05.363405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:05.363442 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:05.364986 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:05.365393 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:05.365396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:05.365397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:05.365444 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:05.365656 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:05.365659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:05.365662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:05.365831 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:05.365954 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:05.366192 (+ 238us) server_negotiation.cc:299] Negotiation successful
0109 21:03:05.366332 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":115,"threads_started":1}
I20250109 21:03:05.369112 14517 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:03:05.369560 14517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:03:05.374367 14657 tablet_service.cc:1467] Processing CreateTablet for tablet 385b9a415d654407a9adec91e5a657d7 (DEFAULT_TABLE table=test-table [id=53739e810dfb47fea81427364d53a0ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:05.374475 14797 tablet_service.cc:1467] Processing CreateTablet for tablet 385b9a415d654407a9adec91e5a657d7 (DEFAULT_TABLE table=test-table [id=53739e810dfb47fea81427364d53a0ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:05.374608 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 385b9a415d654407a9adec91e5a657d7 (DEFAULT_TABLE table=test-table [id=53739e810dfb47fea81427364d53a0ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:05.374696 14657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 385b9a415d654407a9adec91e5a657d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:05.374764 14797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 385b9a415d654407a9adec91e5a657d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:05.374917 14937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 385b9a415d654407a9adec91e5a657d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:05.377104 15127 tablet_bootstrap.cc:492] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5: Bootstrap starting.
I20250109 21:03:05.377554 15125 tablet_bootstrap.cc:492] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1: Bootstrap starting.
I20250109 21:03:05.378054 15127 tablet_bootstrap.cc:654] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:05.378381 15125 tablet_bootstrap.cc:654] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:05.378623 15126 tablet_bootstrap.cc:492] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242: Bootstrap starting.
I20250109 21:03:05.379271 15127 tablet_bootstrap.cc:492] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5: No bootstrap required, opened a new log
I20250109 21:03:05.379367 15127 ts_tablet_manager.cc:1397] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250109 21:03:05.379503 15125 tablet_bootstrap.cc:492] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1: No bootstrap required, opened a new log
I20250109 21:03:05.379587 15125 ts_tablet_manager.cc:1397] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250109 21:03:05.379606 15126 tablet_bootstrap.cc:654] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:05.379827 15127 raft_consensus.cc:357] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:05.379976 15127 raft_consensus.cc:383] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:05.380045 15127 raft_consensus.cc:738] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee46231170a4a3c918e358cff7d3ff5, State: Initialized, Role: FOLLOWER
I20250109 21:03:05.380071 15125 raft_consensus.cc:357] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:05.380231 15125 raft_consensus.cc:383] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:05.380311 15125 raft_consensus.cc:738] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982b8e3e37334d01828202cea3efaba1, State: Initialized, Role: FOLLOWER
I20250109 21:03:05.380250 15127 consensus_queue.cc:260] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:05.380549 15125 consensus_queue.cc:260] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:05.380646 15127 ts_tablet_manager.cc:1428] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250109 21:03:05.380753 15126 tablet_bootstrap.cc:492] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242: No bootstrap required, opened a new log
I20250109 21:03:05.380839 15126 ts_tablet_manager.cc:1397] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:05.381229 15125 ts_tablet_manager.cc:1428] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 21:03:05.381182 15126 raft_consensus.cc:357] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:05.381294 15126 raft_consensus.cc:383] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:05.381328 15126 raft_consensus.cc:738] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221f5fb33e21499cad02660e45d23242, State: Initialized, Role: FOLLOWER
I20250109 21:03:05.381474 15126 consensus_queue.cc:260] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:05.381747 15126 ts_tablet_manager.cc:1428] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 21:03:05.451450 15131 raft_consensus.cc:491] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:03:05.451614 15131 raft_consensus.cc:513] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:05.452086 15131 leader_election.cc:290] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982b8e3e37334d01828202cea3efaba1 (127.12.126.65:42237), 221f5fb33e21499cad02660e45d23242 (127.12.126.67:40451)
I20250109 21:03:05.452607 14677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385b9a415d654407a9adec91e5a657d7" candidate_uuid: "fee46231170a4a3c918e358cff7d3ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982b8e3e37334d01828202cea3efaba1" is_pre_election: true
I20250109 21:03:05.452644 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385b9a415d654407a9adec91e5a657d7" candidate_uuid: "fee46231170a4a3c918e358cff7d3ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "221f5fb33e21499cad02660e45d23242" is_pre_election: true
I20250109 21:03:05.452775 14677 raft_consensus.cc:2463] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fee46231170a4a3c918e358cff7d3ff5 in term 0.
I20250109 21:03:05.452777 14957 raft_consensus.cc:2463] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fee46231170a4a3c918e358cff7d3ff5 in term 0.
I20250109 21:03:05.453147 14749 leader_election.cc:304] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 982b8e3e37334d01828202cea3efaba1, fee46231170a4a3c918e358cff7d3ff5; no voters:
I20250109 21:03:05.453442 15131 raft_consensus.cc:2798] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:03:05.453497 15131 raft_consensus.cc:491] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:03:05.453541 15131 raft_consensus.cc:3054] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:05.454504 15131 raft_consensus.cc:513] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:05.454885 15131 leader_election.cc:290] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 982b8e3e37334d01828202cea3efaba1 (127.12.126.65:42237), 221f5fb33e21499cad02660e45d23242 (127.12.126.67:40451)
I20250109 21:03:05.455308 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385b9a415d654407a9adec91e5a657d7" candidate_uuid: "fee46231170a4a3c918e358cff7d3ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "221f5fb33e21499cad02660e45d23242"
I20250109 21:03:05.455353 14677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385b9a415d654407a9adec91e5a657d7" candidate_uuid: "fee46231170a4a3c918e358cff7d3ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982b8e3e37334d01828202cea3efaba1"
I20250109 21:03:05.455448 14957 raft_consensus.cc:3054] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:05.455485 14677 raft_consensus.cc:3054] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:05.456513 14957 raft_consensus.cc:2463] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fee46231170a4a3c918e358cff7d3ff5 in term 1.
I20250109 21:03:05.456558 14677 raft_consensus.cc:2463] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fee46231170a4a3c918e358cff7d3ff5 in term 1.
I20250109 21:03:05.456830 14749 leader_election.cc:304] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 221f5fb33e21499cad02660e45d23242, fee46231170a4a3c918e358cff7d3ff5; no voters:
I20250109 21:03:05.457029 15131 raft_consensus.cc:2798] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:05.457275 15131 raft_consensus.cc:695] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [term 1 LEADER]: Becoming Leader. State: Replica: fee46231170a4a3c918e358cff7d3ff5, State: Running, Role: LEADER
I20250109 21:03:05.457458 15131 consensus_queue.cc:237] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:05.459187 14517 catalog_manager.cc:5526] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fee46231170a4a3c918e358cff7d3ff5 (127.12.126.66). New cstate: current_term: 1 leader_uuid: "fee46231170a4a3c918e358cff7d3ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } health_report { overall_health: UNKNOWN } } }
I20250109 21:03:05.487946 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:05.485167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46438 (local address 127.12.126.66:46033)
0109 21:03:05.485429 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:05.485432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:05.485444 (+ 12us) server_negotiation.cc:407] Connection header received
0109 21:03:05.485484 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:05.485486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:05.485529 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:05.485604 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:05.485948 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:05.486473 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:05.487243 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:05.487399 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:05.487443 (+ 44us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:05.487515 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:05.487585 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:05.487717 (+ 132us) server_negotiation.cc:299] Negotiation successful
0109 21:03:05.487760 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":102,"threads_started":1}
I20250109 21:03:05.494658 14957 raft_consensus.cc:1270] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242 [term 1 FOLLOWER]: Refusing update from remote peer fee46231170a4a3c918e358cff7d3ff5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:05.494654 14677 raft_consensus.cc:1270] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1 [term 1 FOLLOWER]: Refusing update from remote peer fee46231170a4a3c918e358cff7d3ff5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:05.495086 15132 consensus_queue.cc:1035] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:05.495226 15131 consensus_queue.cc:1035] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:05.528491 14519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39526:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:05.528661 14519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39526:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:05.530680 14519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0: Sending DeleteTablet for 3 replicas of tablet 385b9a415d654407a9adec91e5a657d7
I20250109 21:03:05.531355 14657 tablet_service.cc:1514] Processing DeleteTablet for tablet 385b9a415d654407a9adec91e5a657d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:05 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:40588
I20250109 21:03:05.531378 14797 tablet_service.cc:1514] Processing DeleteTablet for tablet 385b9a415d654407a9adec91e5a657d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:05 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:44552
I20250109 21:03:05.531431 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet 385b9a415d654407a9adec91e5a657d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:05 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:59910
I20250109 21:03:05.531574 15115 tablet_replica.cc:331] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5: stopping tablet replica
I20250109 21:03:05.531596 15114 tablet_replica.cc:331] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1: stopping tablet replica
I20250109 21:03:05.531610 15116 tablet_replica.cc:331] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242: stopping tablet replica
I20250109 21:03:05.531687 15114 raft_consensus.cc:2238] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:05.531687 15115 raft_consensus.cc:2238] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:03:05.531700 15116 raft_consensus.cc:2238] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:05.531777 15114 raft_consensus.cc:2267] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:05.531805 15116 raft_consensus.cc:2267] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:05.531883 15115 raft_consensus.cc:2267] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:05.532646 15115 ts_tablet_manager.cc:1905] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:05.532696 15116 ts_tablet_manager.cc:1905] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:05.532711 14519 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 21:03:05.532922 15114 ts_tablet_manager.cc:1905] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 21:03:05.533146 14519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:03:05.535485 15116 ts_tablet_manager.cc:1918] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250109 21:03:05.535485 15114 ts_tablet_manager.cc:1918] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250109 21:03:05.535567 15116 log.cc:1198] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/385b9a415d654407a9adec91e5a657d7
I20250109 21:03:05.535567 15114 log.cc:1198] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-0/wal/wals/385b9a415d654407a9adec91e5a657d7
I20250109 21:03:05.535831 15114 ts_tablet_manager.cc:1939] T 385b9a415d654407a9adec91e5a657d7 P 982b8e3e37334d01828202cea3efaba1: Deleting consensus metadata
I20250109 21:03:05.535831 15116 ts_tablet_manager.cc:1939] T 385b9a415d654407a9adec91e5a657d7 P 221f5fb33e21499cad02660e45d23242: Deleting consensus metadata
I20250109 21:03:05.536752 14505 catalog_manager.cc:4872] TS 221f5fb33e21499cad02660e45d23242 (127.12.126.67:40451): tablet 385b9a415d654407a9adec91e5a657d7 (table test-table [id=53739e810dfb47fea81427364d53a0ee]) successfully deleted
I20250109 21:03:05.537111 14505 catalog_manager.cc:4872] TS 982b8e3e37334d01828202cea3efaba1 (127.12.126.65:42237): tablet 385b9a415d654407a9adec91e5a657d7 (table test-table [id=53739e810dfb47fea81427364d53a0ee]) successfully deleted
I20250109 21:03:05.537303 15115 ts_tablet_manager.cc:1918] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250109 21:03:05.537379 15115 log.cc:1198] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/385b9a415d654407a9adec91e5a657d7
I20250109 21:03:05.537643 15115 ts_tablet_manager.cc:1939] T 385b9a415d654407a9adec91e5a657d7 P fee46231170a4a3c918e358cff7d3ff5: Deleting consensus metadata
I20250109 21:03:05.538316 14505 catalog_manager.cc:4872] TS fee46231170a4a3c918e358cff7d3ff5 (127.12.126.66:46033): tablet 385b9a415d654407a9adec91e5a657d7 (table test-table [id=53739e810dfb47fea81427364d53a0ee]) successfully deleted
I20250109 21:03:05.539327 14657 tablet_service.cc:1467] Processing CreateTablet for tablet bf65b516a318495eaa4086536597eb87 (DEFAULT_TABLE table=test-table [id=1429ded05e22465db1b864e47ee2e52d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:05.539436 14797 tablet_service.cc:1467] Processing CreateTablet for tablet bf65b516a318495eaa4086536597eb87 (DEFAULT_TABLE table=test-table [id=1429ded05e22465db1b864e47ee2e52d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:05.539626 14657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf65b516a318495eaa4086536597eb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:05.539649 14797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf65b516a318495eaa4086536597eb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:05.539665 14937 tablet_service.cc:1467] Processing CreateTablet for tablet bf65b516a318495eaa4086536597eb87 (DEFAULT_TABLE table=test-table [id=1429ded05e22465db1b864e47ee2e52d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:05.539973 14937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf65b516a318495eaa4086536597eb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:05.541675 15126 tablet_bootstrap.cc:492] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242: Bootstrap starting.
I20250109 21:03:05.541790 15127 tablet_bootstrap.cc:492] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5: Bootstrap starting.
I20250109 21:03:05.542595 15126 tablet_bootstrap.cc:654] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:05.542654 15127 tablet_bootstrap.cc:654] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:05.543207 15125 tablet_bootstrap.cc:492] T bf65b516a318495eaa4086536597eb87 P 982b8e3e37334d01828202cea3efaba1: Bootstrap starting.
I20250109 21:03:05.543877 15127 tablet_bootstrap.cc:492] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5: No bootstrap required, opened a new log
I20250109 21:03:05.543942 15127 ts_tablet_manager.cc:1397] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250109 21:03:05.544030 15125 tablet_bootstrap.cc:654] T bf65b516a318495eaa4086536597eb87 P 982b8e3e37334d01828202cea3efaba1: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:05.544260 15127 raft_consensus.cc:357] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:05.544342 15127 raft_consensus.cc:383] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:05.544363 15127 raft_consensus.cc:738] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee46231170a4a3c918e358cff7d3ff5, State: Initialized, Role: FOLLOWER
I20250109 21:03:05.544495 15127 consensus_queue.cc:260] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:05.544773 15127 ts_tablet_manager.cc:1428] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 21:03:05.545025 15126 tablet_bootstrap.cc:492] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242: No bootstrap required, opened a new log
I20250109 21:03:05.545095 15126 ts_tablet_manager.cc:1397] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250109 21:03:05.545550 15126 raft_consensus.cc:357] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:05.545652 15126 raft_consensus.cc:383] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:05.545691 15126 raft_consensus.cc:738] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221f5fb33e21499cad02660e45d23242, State: Initialized, Role: FOLLOWER
I20250109 21:03:05.545827 15126 consensus_queue.cc:260] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:05.546061 15126 ts_tablet_manager.cc:1428] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 21:03:05.546604 15125 tablet_bootstrap.cc:492] T bf65b516a318495eaa4086536597eb87 P 982b8e3e37334d01828202cea3efaba1: No bootstrap required, opened a new log
I20250109 21:03:05.546707 15125 ts_tablet_manager.cc:1397] T bf65b516a318495eaa4086536597eb87 P 982b8e3e37334d01828202cea3efaba1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20250109 21:03:05.547097 15125 raft_consensus.cc:357] T bf65b516a318495eaa4086536597eb87 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:05.547209 15125 raft_consensus.cc:383] T bf65b516a318495eaa4086536597eb87 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:05.547245 15125 raft_consensus.cc:738] T bf65b516a318495eaa4086536597eb87 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982b8e3e37334d01828202cea3efaba1, State: Initialized, Role: FOLLOWER
I20250109 21:03:05.547386 15125 consensus_queue.cc:260] T bf65b516a318495eaa4086536597eb87 P 982b8e3e37334d01828202cea3efaba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:05.547663 15125 ts_tablet_manager.cc:1428] T bf65b516a318495eaa4086536597eb87 P 982b8e3e37334d01828202cea3efaba1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 21:03:05.674738 15105 raft_consensus.cc:491] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:03:05.674937 15105 raft_consensus.cc:513] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:05.675412 15105 leader_election.cc:290] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982b8e3e37334d01828202cea3efaba1 (127.12.126.65:42237), fee46231170a4a3c918e358cff7d3ff5 (127.12.126.66:46033)
I20250109 21:03:05.675877 14677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf65b516a318495eaa4086536597eb87" candidate_uuid: "221f5fb33e21499cad02660e45d23242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982b8e3e37334d01828202cea3efaba1" is_pre_election: true
I20250109 21:03:05.676023 14677 raft_consensus.cc:2463] T bf65b516a318495eaa4086536597eb87 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 221f5fb33e21499cad02660e45d23242 in term 0.
I20250109 21:03:05.675997 14817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf65b516a318495eaa4086536597eb87" candidate_uuid: "221f5fb33e21499cad02660e45d23242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee46231170a4a3c918e358cff7d3ff5" is_pre_election: true
I20250109 21:03:05.676231 14817 raft_consensus.cc:2463] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 221f5fb33e21499cad02660e45d23242 in term 0.
I20250109 21:03:05.676383 14889 leader_election.cc:304] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 221f5fb33e21499cad02660e45d23242, 982b8e3e37334d01828202cea3efaba1; no voters:
I20250109 21:03:05.676574 15105 raft_consensus.cc:2798] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:03:05.676637 15105 raft_consensus.cc:491] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:03:05.676688 15105 raft_consensus.cc:3054] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:05.677704 15105 raft_consensus.cc:513] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:05.678020 15105 leader_election.cc:290] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [CANDIDATE]: Term 1 election: Requested vote from peers 982b8e3e37334d01828202cea3efaba1 (127.12.126.65:42237), fee46231170a4a3c918e358cff7d3ff5 (127.12.126.66:46033)
I20250109 21:03:05.678370 14677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf65b516a318495eaa4086536597eb87" candidate_uuid: "221f5fb33e21499cad02660e45d23242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982b8e3e37334d01828202cea3efaba1"
I20250109 21:03:05.678440 14817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf65b516a318495eaa4086536597eb87" candidate_uuid: "221f5fb33e21499cad02660e45d23242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee46231170a4a3c918e358cff7d3ff5"
I20250109 21:03:05.678512 14677 raft_consensus.cc:3054] T bf65b516a318495eaa4086536597eb87 P 982b8e3e37334d01828202cea3efaba1 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:05.678579 14817 raft_consensus.cc:3054] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:05.679632 14817 raft_consensus.cc:2463] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 221f5fb33e21499cad02660e45d23242 in term 1.
I20250109 21:03:05.679632 14677 raft_consensus.cc:2463] T bf65b516a318495eaa4086536597eb87 P 982b8e3e37334d01828202cea3efaba1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 221f5fb33e21499cad02660e45d23242 in term 1.
I20250109 21:03:05.679919 14889 leader_election.cc:304] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 221f5fb33e21499cad02660e45d23242, 982b8e3e37334d01828202cea3efaba1; no voters:
I20250109 21:03:05.680104 15105 raft_consensus.cc:2798] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:05.680228 15105 raft_consensus.cc:695] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [term 1 LEADER]: Becoming Leader. State: Replica: 221f5fb33e21499cad02660e45d23242, State: Running, Role: LEADER
I20250109 21:03:05.680419 15105 consensus_queue.cc:237] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } }
I20250109 21:03:05.682106 14517 catalog_manager.cc:5526] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 reported cstate change: term changed from 0 to 1, leader changed from <none> to 221f5fb33e21499cad02660e45d23242 (127.12.126.67). New cstate: current_term: 1 leader_uuid: "221f5fb33e21499cad02660e45d23242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "221f5fb33e21499cad02660e45d23242" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40451 } health_report { overall_health: HEALTHY } } }
I20250109 21:03:05.705763 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:05.703225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.12.126.67:40451)
0109 21:03:05.703518 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:05.703521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:05.703531 (+ 10us) server_negotiation.cc:407] Connection header received
0109 21:03:05.703567 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:05.703569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:05.703611 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:05.703690 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:05.704027 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:05.704520 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:05.705108 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:05.705283 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:05.705328 (+ 45us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:05.705391 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:05.705468 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:05.705567 (+ 99us) server_negotiation.cc:299] Negotiation successful
0109 21:03:05.705612 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
I20250109 21:03:05.711692 14817 raft_consensus.cc:1270] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5 [term 1 FOLLOWER]: Refusing update from remote peer 221f5fb33e21499cad02660e45d23242: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:05.711743 14677 raft_consensus.cc:1270] T bf65b516a318495eaa4086536597eb87 P 982b8e3e37334d01828202cea3efaba1 [term 1 FOLLOWER]: Refusing update from remote peer 221f5fb33e21499cad02660e45d23242: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:05.712103 15105 consensus_queue.cc:1035] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982b8e3e37334d01828202cea3efaba1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:05.712232 15091 consensus_queue.cc:1035] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fee46231170a4a3c918e358cff7d3ff5" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 21:03:05.727948 15096 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20250109 21:03:07.278630 14520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39526:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:07.278782 14520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39526:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:07.281003 14520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f27d668791c44620931035e3f3d7d7f0: Sending DeleteTablet for 3 replicas of tablet bf65b516a318495eaa4086536597eb87
I20250109 21:03:07.281533 14657 tablet_service.cc:1514] Processing DeleteTablet for tablet bf65b516a318495eaa4086536597eb87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:07 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:40588
I20250109 21:03:07.281591 14797 tablet_service.cc:1514] Processing DeleteTablet for tablet bf65b516a318495eaa4086536597eb87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:07 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:44552
I20250109 21:03:07.281649 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet bf65b516a318495eaa4086536597eb87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:07 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:59910
I20250109 21:03:07.281931 15172 tablet_replica.cc:331] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242: stopping tablet replica
I20250109 21:03:07.281972 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 14592
I20250109 21:03:07.282943 15172 raft_consensus.cc:2238] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:03:07.283174 15172 raft_consensus.cc:2267] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:07.283430 15171 tablet_replica.cc:331] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5: stopping tablet replica
I20250109 21:03:07.283567 15171 raft_consensus.cc:2238] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:07.283684 15171 raft_consensus.cc:2267] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:07.284242 15172 ts_tablet_manager.cc:1905] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:07.286234 15172 ts_tablet_manager.cc:1918] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250109 21:03:07.286319 15172 log.cc:1198] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/bf65b516a318495eaa4086536597eb87
I20250109 21:03:07.286412 15171 ts_tablet_manager.cc:1905] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:07.286675 15172 ts_tablet_manager.cc:1939] T bf65b516a318495eaa4086536597eb87 P 221f5fb33e21499cad02660e45d23242: Deleting consensus metadata
I20250109 21:03:07.287468 14505 catalog_manager.cc:4872] TS 221f5fb33e21499cad02660e45d23242 (127.12.126.67:40451): tablet bf65b516a318495eaa4086536597eb87 (table test-table [id=1429ded05e22465db1b864e47ee2e52d]) successfully deleted
I20250109 21:03:07.288353 15171 ts_tablet_manager.cc:1918] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250109 21:03:07.288417 15171 log.cc:1198] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/bf65b516a318495eaa4086536597eb87
I20250109 21:03:07.288663 15171 ts_tablet_manager.cc:1939] T bf65b516a318495eaa4086536597eb87 P fee46231170a4a3c918e358cff7d3ff5: Deleting consensus metadata
I20250109 21:03:07.289254 14505 catalog_manager.cc:4872] TS fee46231170a4a3c918e358cff7d3ff5 (127.12.126.66:46033): tablet bf65b516a318495eaa4086536597eb87 (table test-table [id=1429ded05e22465db1b864e47ee2e52d]) successfully deleted
W20250109 21:03:07.290057 14505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.126.65:42237 (error 108)
I20250109 21:03:07.290328 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 14732
I20250109 21:03:07.291543 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:07.291156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:42237 (local address 127.0.0.1:47396)
0109 21:03:07.291377 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:07.291458 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.65:42237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
W20250109 21:03:07.291649 14505 catalog_manager.cc:4615] TS 982b8e3e37334d01828202cea3efaba1 (127.12.126.65:42237): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf65b516a318495eaa4086536597eb87: Network error: Client connection negotiation failed: client connection to 127.12.126.65:42237: connect: Connection refused (error 111)
I20250109 21:03:07.298471 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 14872
I20250109 21:03:07.305176 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 14488
2025-01-09T21:03:07Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6383 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:03:07 dist-test-slave-g0zb krb5kdc[15177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:03:07 dist-test-slave-g0zb krb5kdc[15177](info): set up 2 sockets
Jan 09 21:03:07 dist-test-slave-g0zb krb5kdc[15177](info): commencing operation
krb5kdc: starting...
W20250109 21:03:08.335500 12793 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:03:08 dist-test-slave-g0zb krb5kdc[15177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:03:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:03:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:03:08.454038 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:44159
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:44159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:08.560591 15190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:08.560838 15190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:08.560917 15190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:08.564081 15190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:08.564148 15190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:08.564180 15190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:08.564209 15190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:08.564267 15190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:03:08.568631 15190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:44159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:44159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:08.569804 15190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:08.570675 15190 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:08.577354 15198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:08.578068 15196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:08.578904 15195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:08.579799 15190 server_base.cc:1034] running on GCE node
I20250109 21:03:08.580317 15190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:08.581238 15190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:08.582398 15190 hybrid_clock.cc:648] HybridClock initialized: now 1736456588582385 us; error 27 us; skew 500 ppm
Jan 09 21:03:08 dist-test-slave-g0zb krb5kdc[15177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456588, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:08.584954 15190 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:03:08.585851 15190 webserver.cc:458] Webserver started at http://127.12.126.126:33455/ using document root <none> and password file <none>
I20250109 21:03:08.586395 15190 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:08.586458 15190 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:08.586638 15190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:08.588492 15190 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "b84cf4cc71fe4be8856267430948bca1"
format_stamp: "Formatted at 2025-01-09 21:03:08 on dist-test-slave-g0zb"
server_key: "f418274ba7308556e08cea06a728b536"
server_key_iv: "702b276a59a1195f44c7441814e37837"
server_key_version: "encryptionkey@0"
I20250109 21:03:08.588991 15190 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "b84cf4cc71fe4be8856267430948bca1"
format_stamp: "Formatted at 2025-01-09 21:03:08 on dist-test-slave-g0zb"
server_key: "f418274ba7308556e08cea06a728b536"
server_key_iv: "702b276a59a1195f44c7441814e37837"
server_key_version: "encryptionkey@0"
I20250109 21:03:08.592360 15190 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20250109 21:03:08.594602 15205 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:08.595702 15190 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:08.595815 15190 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "b84cf4cc71fe4be8856267430948bca1"
format_stamp: "Formatted at 2025-01-09 21:03:08 on dist-test-slave-g0zb"
server_key: "f418274ba7308556e08cea06a728b536"
server_key_iv: "702b276a59a1195f44c7441814e37837"
server_key_version: "encryptionkey@0"
I20250109 21:03:08.595904 15190 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:08.609061 15190 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:08.611846 15190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:08.612013 15190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:08.618541 15190 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:44159
I20250109 21:03:08.618779 15257 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:44159 every 8 connection(s)
I20250109 21:03:08.619560 15190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:03:08.622045 15258 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:08.627031 15258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1: Bootstrap starting.
I20250109 21:03:08.629272 15258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:08.629709 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 15190
I20250109 21:03:08.629819 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:03:08.630046 12793 external_mini_cluster.cc:1453] Setting key de320d618d1aaf7ccaa6c02c8d029f1c
I20250109 21:03:08.630040 15258 log.cc:826] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:08.631947 15258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1: No bootstrap required, opened a new log
I20250109 21:03:08.634403 15258 raft_consensus.cc:357] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84cf4cc71fe4be8856267430948bca1" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 44159 } }
I20250109 21:03:08.634586 15258 raft_consensus.cc:383] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:08.634634 15258 raft_consensus.cc:738] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b84cf4cc71fe4be8856267430948bca1, State: Initialized, Role: FOLLOWER
I20250109 21:03:08.635066 15258 consensus_queue.cc:260] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84cf4cc71fe4be8856267430948bca1" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 44159 } }
I20250109 21:03:08.635201 15258 raft_consensus.cc:397] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:03:08.635270 15258 raft_consensus.cc:491] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:03:08.635378 15258 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:08.636389 15258 raft_consensus.cc:513] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84cf4cc71fe4be8856267430948bca1" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 44159 } }
I20250109 21:03:08.636708 15258 leader_election.cc:304] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b84cf4cc71fe4be8856267430948bca1; no voters:
I20250109 21:03:08.636986 15258 leader_election.cc:290] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:03:08.637101 15263 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:08.637279 15263 raft_consensus.cc:695] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1 [term 1 LEADER]: Becoming Leader. State: Replica: b84cf4cc71fe4be8856267430948bca1, State: Running, Role: LEADER
Jan 09 21:03:08 dist-test-slave-g0zb krb5kdc[15177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:08.637653 15263 consensus_queue.cc:237] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84cf4cc71fe4be8856267430948bca1" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 44159 } }
I20250109 21:03:08.638157 15258 sys_catalog.cc:564] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:08.638945 15265 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b84cf4cc71fe4be8856267430948bca1. Latest consensus state: current_term: 1 leader_uuid: "b84cf4cc71fe4be8856267430948bca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84cf4cc71fe4be8856267430948bca1" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 44159 } } }
I20250109 21:03:08.639225 15265 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:08.639701 15264 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b84cf4cc71fe4be8856267430948bca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84cf4cc71fe4be8856267430948bca1" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 44159 } } }
I20250109 21:03:08.639808 15264 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:08.639748 15272 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:03:08.643745 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:08.631074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.12.126.126:44159)
0109 21:03:08.631462 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:08.631472 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:08.631519 (+ 47us) server_negotiation.cc:407] Connection header received
0109 21:03:08.632121 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:08.632140 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:08.632419 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:08.632760 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:08.633652 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:08.634857 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:08.635560 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:08.635851 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:08.638261 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:08.638282 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:08.638297 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:08.638341 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:08.641006 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:08.641542 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:08.641547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:08.641553 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:08.641645 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:08.641915 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:08.641921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:08.641923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:08.642361 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:08.642586 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:08.642889 (+ 303us) server_negotiation.cc:299] Negotiation successful
0109 21:03:08.643151 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":120,"threads_started":1}
I20250109 21:03:08.644068 15272 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:03:08.649360 15272 catalog_manager.cc:1348] Generated new cluster ID: 593f6a9e19bc4f39b17e98e3a137eff5
I20250109 21:03:08.649443 15272 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:03:08.662307 15272 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:03:08.663412 15272 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:03:08.674504 15272 catalog_manager.cc:5899] T 00000000000000000000000000000000 P b84cf4cc71fe4be8856267430948bca1: Generated new TSK 0
I20250109 21:03:08.675292 15272 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
I20250109 21:03:08.703498 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:44159
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:08.808531 15284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:08.808743 15284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:08.808815 15284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:08.811911 15284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:08.811970 15284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:08.812075 15284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:03:08.816416 15284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:44159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:08.817723 15284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:08.818758 15284 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:08.827044 15289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:08.827044 15292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:08.827883 15290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:08.828641 15284 server_base.cc:1034] running on GCE node
I20250109 21:03:08.828994 15284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:08.829535 15284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:08.830703 15284 hybrid_clock.cc:648] HybridClock initialized: now 1736456588830681 us; error 33 us; skew 500 ppm
Jan 09 21:03:08 dist-test-slave-g0zb krb5kdc[15177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456588, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:08.833281 15284 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:03:08.834153 15284 webserver.cc:458] Webserver started at http://127.12.126.65:36195/ using document root <none> and password file <none>
I20250109 21:03:08.834700 15284 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:08.834761 15284 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:08.834976 15284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:08.836747 15284 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "7e4aa35f33944702835cfa574085feea"
format_stamp: "Formatted at 2025-01-09 21:03:08 on dist-test-slave-g0zb"
server_key: "caaee4852ace9790500077d2540d12ca"
server_key_iv: "484a1e3dd6ce51ad2cea24fdd8f40ae3"
server_key_version: "encryptionkey@0"
I20250109 21:03:08.837235 15284 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "7e4aa35f33944702835cfa574085feea"
format_stamp: "Formatted at 2025-01-09 21:03:08 on dist-test-slave-g0zb"
server_key: "caaee4852ace9790500077d2540d12ca"
server_key_iv: "484a1e3dd6ce51ad2cea24fdd8f40ae3"
server_key_version: "encryptionkey@0"
I20250109 21:03:08.840544 15284 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:03:08.842523 15299 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:08.843566 15284 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:08.843672 15284 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "7e4aa35f33944702835cfa574085feea"
format_stamp: "Formatted at 2025-01-09 21:03:08 on dist-test-slave-g0zb"
server_key: "caaee4852ace9790500077d2540d12ca"
server_key_iv: "484a1e3dd6ce51ad2cea24fdd8f40ae3"
server_key_version: "encryptionkey@0"
I20250109 21:03:08.843765 15284 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:08.883129 15284 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:08.885987 15284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:08.886179 15284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:08.886734 15284 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:08.887610 15284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:08.887665 15284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:08.887725 15284 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:08.887758 15284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:08.895908 15284 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:35491
I20250109 21:03:08.895938 15412 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:35491 every 8 connection(s)
I20250109 21:03:08.896800 15284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:03:08.899361 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 15284
I20250109 21:03:08.899480 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:03:08.899761 12793 external_mini_cluster.cc:1453] Setting key e084ceaf00e4bdba7a2a5df87e2738e0
Jan 09 21:03:08 dist-test-slave-g0zb krb5kdc[15177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456588, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.126.66@KRBTEST.COM; defaulting to no policy
I20250109 21:03:08.910396 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:08.898442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:40209 (local address 127.12.126.126:44159)
0109 21:03:08.898617 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:08.898622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:08.899385 (+ 763us) server_negotiation.cc:407] Connection header received
0109 21:03:08.900895 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:08.900899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:08.900944 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:08.901033 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:08.902753 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:08.903319 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:08.904240 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:08.904358 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:08.906885 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:08.906907 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:08.906913 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:08.906948 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:08.908624 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:08.909073 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:08.909077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:08.909079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:08.909132 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:08.909388 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:08.909393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:08.909398 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:08.909624 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:08.909723 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:08.910146 (+ 423us) server_negotiation.cc:299] Negotiation successful
0109 21:03:08.910251 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20250109 21:03:08.911188 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:08.898750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:44159 (local address 127.12.126.65:40209)
0109 21:03:08.899216 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:08.899271 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:08.900697 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:08.901207 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:08.901216 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:08.901774 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:08.902560 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:08.902581 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:08.903446 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:08.903451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:08.904110 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:08.904124 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:08.904363 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:08.905020 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:08.905043 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:08.906703 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:08.908736 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:08.908741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:08.908750 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:08.908974 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:08.909220 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:08.909222 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:08.909223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:08.909297 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:08.909726 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:08.909731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:08.909950 (+ 219us) client_negotiation.cc:769] Sending connection context
0109 21:03:08.910111 (+ 161us) client_negotiation.cc:240] Negotiation successful
0109 21:03:08.910401 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.66@KRBTEST.COM" created.
I20250109 21:03:08.912374 15413 heartbeater.cc:346] Connected to a master server at 127.12.126.126:44159
I20250109 21:03:08.912600 15413 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:08.913090 15413 heartbeater.cc:510] Master 127.12.126.126:44159 requested a full tablet report, sending...
I20250109 21:03:08.914661 15221 ts_manager.cc:194] Registered new tserver with Master: 7e4aa35f33944702835cfa574085feea (127.12.126.65:35491)
I20250109 21:03:08.915824 15221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:40209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
I20250109 21:03:08.928181 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:44159
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:09.035840 15418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:09.036057 15418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:09.036113 15418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:09.039198 15418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:09.039258 15418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:09.039335 15418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:03:09.043546 15418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:44159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:09.044510 15418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:09.045297 15418 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:09.052899 15426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:09.053638 15424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:09.054261 15423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:09.055166 15418 server_base.cc:1034] running on GCE node
I20250109 21:03:09.055548 15418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:09.056056 15418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:09.057236 15418 hybrid_clock.cc:648] HybridClock initialized: now 1736456589057216 us; error 25 us; skew 500 ppm
Jan 09 21:03:09 dist-test-slave-g0zb krb5kdc[15177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456589, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:09.059945 15418 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:03:09.060809 15418 webserver.cc:458] Webserver started at http://127.12.126.66:33277/ using document root <none> and password file <none>
I20250109 21:03:09.061360 15418 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:09.061420 15418 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:09.061614 15418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:09.063458 15418 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "055a75cbd07e408d9d7583a2d1a5f3e4"
format_stamp: "Formatted at 2025-01-09 21:03:09 on dist-test-slave-g0zb"
server_key: "267343512e092d6f33c0ff5d4b6f7b99"
server_key_iv: "276a99942e4c352849822a36b5370d77"
server_key_version: "encryptionkey@0"
I20250109 21:03:09.063949 15418 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "055a75cbd07e408d9d7583a2d1a5f3e4"
format_stamp: "Formatted at 2025-01-09 21:03:09 on dist-test-slave-g0zb"
server_key: "267343512e092d6f33c0ff5d4b6f7b99"
server_key_iv: "276a99942e4c352849822a36b5370d77"
server_key_version: "encryptionkey@0"
I20250109 21:03:09.067245 15418 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:03:09.069413 15433 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:09.070349 15418 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:09.070461 15418 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "055a75cbd07e408d9d7583a2d1a5f3e4"
format_stamp: "Formatted at 2025-01-09 21:03:09 on dist-test-slave-g0zb"
server_key: "267343512e092d6f33c0ff5d4b6f7b99"
server_key_iv: "276a99942e4c352849822a36b5370d77"
server_key_version: "encryptionkey@0"
I20250109 21:03:09.070544 15418 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:09.111397 15418 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:09.114457 15418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:09.114625 15418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:09.115157 15418 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:09.115932 15418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:09.115980 15418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:09.116043 15418 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:09.116070 15418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:09.124287 15418 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:41543
I20250109 21:03:09.124327 15546 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:41543 every 8 connection(s)
I20250109 21:03:09.125216 15418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
Jan 09 21:03:09 dist-test-slave-g0zb krb5kdc[15177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456589, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:09.134733 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 15418
I20250109 21:03:09.134889 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:03:09.135190 12793 external_mini_cluster.cc:1453] Setting key 0c59697b0423074519ead577614551b3
I20250109 21:03:09.137413 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:09.126859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:33657 (local address 127.12.126.126:44159)
0109 21:03:09.127013 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:09.127018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:09.127715 (+ 697us) server_negotiation.cc:407] Connection header received
0109 21:03:09.128866 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:09.128869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:09.128919 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:09.128993 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:09.130203 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:09.130739 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:09.131412 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:09.131556 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:09.133615 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:09.133632 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:09.133634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:09.133657 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:09.135218 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:09.135763 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:09.135766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:09.135768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:09.135821 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:09.136155 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:09.136166 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:09.136167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:09.136334 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:09.136449 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:09.137026 (+ 577us) server_negotiation.cc:299] Negotiation successful
0109 21:03:09.137167 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250109 21:03:09.137864 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:09.127146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:44159 (local address 127.12.126.66:33657)
0109 21:03:09.127570 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:09.127603 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:09.128673 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:09.129130 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:09.129137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:09.129536 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:09.130082 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:09.130095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:09.130905 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:09.130909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:09.131312 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:09.131321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:09.131490 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:09.132046 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:09.132066 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:09.133462 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:09.135336 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:09.135341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:09.135350 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:09.135629 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:09.135911 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:09.135914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:09.135918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:09.136032 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:09.136464 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:09.136472 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:09.136764 (+ 292us) client_negotiation.cc:769] Sending connection context
0109 21:03:09.136936 (+ 172us) client_negotiation.cc:240] Negotiation successful
0109 21:03:09.137158 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
I20250109 21:03:09.139107 15547 heartbeater.cc:346] Connected to a master server at 127.12.126.126:44159
I20250109 21:03:09.139343 15547 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:09.139953 15547 heartbeater.cc:510] Master 127.12.126.126:44159 requested a full tablet report, sending...
I20250109 21:03:09.140973 15221 ts_manager.cc:194] Registered new tserver with Master: 055a75cbd07e408d9d7583a2d1a5f3e4 (127.12.126.66:41543)
I20250109 21:03:09.141516 15221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:33657
WARNING: no policy specified for kudu/127.12.126.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
I20250109 21:03:09.159866 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:44159
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:09.264803 15552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:09.265019 15552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:09.265085 15552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:09.268321 15552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:09.268384 15552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:09.268482 15552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:03:09.272653 15552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:44159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:09.273697 15552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:09.274552 15552 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:09.282255 15558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:09.283071 15552 server_base.cc:1034] running on GCE node
W20250109 21:03:09.283833 15560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:09.284426 15557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:09.284793 15552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:09.285334 15552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:09.286476 15552 hybrid_clock.cc:648] HybridClock initialized: now 1736456589286465 us; error 33 us; skew 500 ppm
Jan 09 21:03:09 dist-test-slave-g0zb krb5kdc[15177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456589, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:09.289065 15552 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:03:09.289983 15552 webserver.cc:458] Webserver started at http://127.12.126.67:38277/ using document root <none> and password file <none>
I20250109 21:03:09.290545 15552 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:09.290603 15552 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:09.290784 15552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:09.292595 15552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "bcf1dc13d3694825b4d2f3ed49c2ed1b"
format_stamp: "Formatted at 2025-01-09 21:03:09 on dist-test-slave-g0zb"
server_key: "3025e9764a2a3130de2cef8566c15ca2"
server_key_iv: "53c218b993e6f8d99c48ce0795af05a7"
server_key_version: "encryptionkey@0"
I20250109 21:03:09.293051 15552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "bcf1dc13d3694825b4d2f3ed49c2ed1b"
format_stamp: "Formatted at 2025-01-09 21:03:09 on dist-test-slave-g0zb"
server_key: "3025e9764a2a3130de2cef8566c15ca2"
server_key_iv: "53c218b993e6f8d99c48ce0795af05a7"
server_key_version: "encryptionkey@0"
I20250109 21:03:09.296425 15552 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20250109 21:03:09.298430 15567 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:09.299403 15552 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:09.299499 15552 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "bcf1dc13d3694825b4d2f3ed49c2ed1b"
format_stamp: "Formatted at 2025-01-09 21:03:09 on dist-test-slave-g0zb"
server_key: "3025e9764a2a3130de2cef8566c15ca2"
server_key_iv: "53c218b993e6f8d99c48ce0795af05a7"
server_key_version: "encryptionkey@0"
I20250109 21:03:09.299582 15552 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:09.337297 15552 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:09.339946 15552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:09.340113 15552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:09.340618 15552 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:09.341373 15552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:09.341418 15552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:09.341473 15552 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:09.341499 15552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:09.349570 15552 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:38911
I20250109 21:03:09.349597 15680 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:38911 every 8 connection(s)
I20250109 21:03:09.350471 15552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:03:09.355437 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 15552
I20250109 21:03:09.355548 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:03:09.355791 12793 external_mini_cluster.cc:1453] Setting key 1a0fc35c60001b1af406c5af4ceb7688
Jan 09 21:03:09 dist-test-slave-g0zb krb5kdc[15177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456589, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:09.361946 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:09.352121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:42731 (local address 127.12.126.126:44159)
0109 21:03:09.352262 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:09.352267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:09.352903 (+ 636us) server_negotiation.cc:407] Connection header received
0109 21:03:09.354060 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:09.354063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:09.354113 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:09.354189 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:09.355506 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:09.356063 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:09.356709 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:09.356886 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:09.358980 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:09.358994 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:09.358996 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:09.359020 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:09.360259 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:09.360701 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:09.360705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:09.360707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:09.360751 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:09.360991 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:09.360995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:09.360996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:09.361135 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:09.361223 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:09.361670 (+ 447us) server_negotiation.cc:299] Negotiation successful
0109 21:03:09.361783 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250109 21:03:09.362550 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:09.352379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:44159 (local address 127.12.126.67:42731)
0109 21:03:09.352773 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:09.352805 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:09.353877 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:09.354364 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:09.354372 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:09.354734 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:09.355344 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:09.355357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:09.356179 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:09.356182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:09.356578 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:09.356604 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:09.356806 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:09.357428 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:09.357451 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:09.358837 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:09.360365 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:09.360370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:09.360382 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:09.360609 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:09.360830 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:09.360833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:09.360835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:09.360911 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:09.361240 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:09.361246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:09.361455 (+ 209us) client_negotiation.cc:769] Sending connection context
0109 21:03:09.361624 (+ 169us) client_negotiation.cc:240] Negotiation successful
0109 21:03:09.361878 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
I20250109 21:03:09.363524 15681 heartbeater.cc:346] Connected to a master server at 127.12.126.126:44159
I20250109 21:03:09.363741 15681 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:09.364281 15681 heartbeater.cc:510] Master 127.12.126.126:44159 requested a full tablet report, sending...
I20250109 21:03:09.365239 15221 ts_manager.cc:194] Registered new tserver with Master: bcf1dc13d3694825b4d2f3ed49c2ed1b (127.12.126.67:38911)
I20250109 21:03:09.365862 15221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:42731
I20250109 21:03:09.369468 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250109 21:03:09.381000 15262 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:09.379656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60278 (local address 127.12.126.126:44159)
0109 21:03:09.379809 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:09.379815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:09.379897 (+ 82us) server_negotiation.cc:407] Connection header received
0109 21:03:09.380035 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:09.380039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:09.380090 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:09.380208 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:09.380901 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60278: BlockingRecv error: recv got EOF from 127.0.0.1:60278 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20250109 21:03:09.382112 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 15284
I20250109 21:03:09.387701 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 15418
I20250109 21:03:09.393175 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 15552
I20250109 21:03:09.398401 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 15190
2025-01-09T21:03:09Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2091 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:03:09 dist-test-slave-g0zb krb5kdc[15693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:03:09 dist-test-slave-g0zb krb5kdc[15693](info): set up 2 sockets
Jan 09 21:03:09 dist-test-slave-g0zb krb5kdc[15693](info): commencing operation
krb5kdc: starting...
W20250109 21:03:10.415448 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:03:10 dist-test-slave-g0zb krb5kdc[15693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:03:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:03:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:03:10.534106 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:41747
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:45353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:41747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:10.637657 15706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:10.637923 15706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:10.638008 15706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:10.641096 15706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:10.641160 15706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:10.641196 15706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:10.641230 15706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:10.641274 15706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:03:10.645319 15706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:45353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:41747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:41747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:10.646349 15706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:10.647207 15706 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:10.654762 15711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:10.655339 15714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:10.655485 15712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:10.656381 15706 server_base.cc:1034] running on GCE node
I20250109 21:03:10.656868 15706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:10.657756 15706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:10.658860 15706 hybrid_clock.cc:648] HybridClock initialized: now 1736456590658851 us; error 23 us; skew 500 ppm
Jan 09 21:03:10 dist-test-slave-g0zb krb5kdc[15693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456590, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:10.666178 15706 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:03:10.667204 15706 webserver.cc:458] Webserver started at http://127.12.126.126:41427/ using document root <none> and password file <none>
I20250109 21:03:10.667768 15706 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:10.667824 15706 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:10.668015 15706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:10.669847 15706 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "2b7cb01dc9274103940a58696cd471ba"
format_stamp: "Formatted at 2025-01-09 21:03:10 on dist-test-slave-g0zb"
server_key: "01858c7c4242485c5422d9f727bc59ed"
server_key_iv: "eb360f34790fa9566f5b567b6046a107"
server_key_version: "encryptionkey@0"
I20250109 21:03:10.670317 15706 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "2b7cb01dc9274103940a58696cd471ba"
format_stamp: "Formatted at 2025-01-09 21:03:10 on dist-test-slave-g0zb"
server_key: "01858c7c4242485c5422d9f727bc59ed"
server_key_iv: "eb360f34790fa9566f5b567b6046a107"
server_key_version: "encryptionkey@0"
I20250109 21:03:10.673610 15706 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250109 21:03:10.675709 15721 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:10.676661 15706 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:10.676757 15706 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "2b7cb01dc9274103940a58696cd471ba"
format_stamp: "Formatted at 2025-01-09 21:03:10 on dist-test-slave-g0zb"
server_key: "01858c7c4242485c5422d9f727bc59ed"
server_key_iv: "eb360f34790fa9566f5b567b6046a107"
server_key_version: "encryptionkey@0"
I20250109 21:03:10.676846 15706 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:10.719697 15706 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:10.722851 15706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:10.723028 15706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:10.729485 15706 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:41747
I20250109 21:03:10.729730 15773 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:41747 every 8 connection(s)
I20250109 21:03:10.730473 15706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:03:10.732939 15774 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:10.737787 15774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba: Bootstrap starting.
I20250109 21:03:10.739601 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 15706
I20250109 21:03:10.739699 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:03:10.739920 12793 external_mini_cluster.cc:1453] Setting key 2bafa656686862767e08f3dd0d9673c7
I20250109 21:03:10.740171 15774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:10.740999 15774 log.cc:826] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:10.742899 15774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba: No bootstrap required, opened a new log
I20250109 21:03:10.746606 15774 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7cb01dc9274103940a58696cd471ba" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41747 } }
I20250109 21:03:10.746886 15774 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:10.746949 15774 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7cb01dc9274103940a58696cd471ba, State: Initialized, Role: FOLLOWER
I20250109 21:03:10.747412 15774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7cb01dc9274103940a58696cd471ba" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41747 } }
I20250109 21:03:10.747539 15774 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:03:10.747599 15774 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:03:10.747699 15774 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba [term 0 FOLLOWER]: Advancing to term 1
Jan 09 21:03:10 dist-test-slave-g0zb krb5kdc[15693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:10.748520 15774 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7cb01dc9274103940a58696cd471ba" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41747 } }
I20250109 21:03:10.748837 15774 leader_election.cc:304] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b7cb01dc9274103940a58696cd471ba; no voters:
I20250109 21:03:10.749109 15774 leader_election.cc:290] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:03:10.749543 15779 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:10.749804 15779 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba [term 1 LEADER]: Becoming Leader. State: Replica: 2b7cb01dc9274103940a58696cd471ba, State: Running, Role: LEADER
I20250109 21:03:10.750095 15779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7cb01dc9274103940a58696cd471ba" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41747 } }
I20250109 21:03:10.750428 15774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:10.751643 15780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b7cb01dc9274103940a58696cd471ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7cb01dc9274103940a58696cd471ba" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41747 } } }
I20250109 21:03:10.751750 15780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:10.752281 15788 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:03:10.752987 15781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b7cb01dc9274103940a58696cd471ba. Latest consensus state: current_term: 1 leader_uuid: "2b7cb01dc9274103940a58696cd471ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7cb01dc9274103940a58696cd471ba" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41747 } } }
I20250109 21:03:10.753115 15781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:10.753863 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:10.740912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.12.126.126:41747)
0109 21:03:10.741324 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:10.741333 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:10.741374 (+ 41us) server_negotiation.cc:407] Connection header received
0109 21:03:10.742063 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:10.742084 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:10.742387 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:10.742689 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:10.743793 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:10.744582 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:10.745543 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:10.745723 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:10.748323 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:10.748346 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:10.748356 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:10.748390 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:10.750528 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:10.751009 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:10.751015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:10.751021 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:10.751118 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:10.751438 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:10.751443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:10.751445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:10.751926 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:10.752154 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:10.752892 (+ 738us) server_negotiation.cc:299] Negotiation successful
0109 21:03:10.753146 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":109,"threads_started":1}
I20250109 21:03:10.755957 15788 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:03:10.761159 15788 catalog_manager.cc:1348] Generated new cluster ID: b497bd007e454faf95c25a993d83661d
I20250109 21:03:10.761214 15788 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:03:10.802004 15788 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:03:10.803382 15788 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:03:10.813278 15788 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 2b7cb01dc9274103940a58696cd471ba: Generated new TSK 0
I20250109 21:03:10.814003 15788 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
I20250109 21:03:10.854274 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:41747
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:45353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:10.964700 15800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:10.964979 15800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:10.965067 15800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:10.968199 15800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:10.968276 15800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:10.968389 15800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:03:10.972764 15800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:45353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:41747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:10.973894 15800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:10.974761 15800 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:10.983193 15805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:10.983263 15806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:10.984156 15808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:10.985013 15800 server_base.cc:1034] running on GCE node
I20250109 21:03:10.985419 15800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:10.985975 15800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:10.987155 15800 hybrid_clock.cc:648] HybridClock initialized: now 1736456590987136 us; error 42 us; skew 500 ppm
Jan 09 21:03:10 dist-test-slave-g0zb krb5kdc[15693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456590, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:10.990222 15800 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:03:10.991297 15800 webserver.cc:458] Webserver started at http://127.12.126.65:32979/ using document root <none> and password file <none>
I20250109 21:03:10.991883 15800 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:10.991954 15800 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:10.992151 15800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:10.994102 15800 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "367003ed9c524fa093732e9027c566d1"
format_stamp: "Formatted at 2025-01-09 21:03:10 on dist-test-slave-g0zb"
server_key: "b448a3e854a87f66494a1d998436fcad"
server_key_iv: "5768a789165f19a4756a251fdd9fb5aa"
server_key_version: "encryptionkey@0"
I20250109 21:03:10.994650 15800 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "367003ed9c524fa093732e9027c566d1"
format_stamp: "Formatted at 2025-01-09 21:03:10 on dist-test-slave-g0zb"
server_key: "b448a3e854a87f66494a1d998436fcad"
server_key_iv: "5768a789165f19a4756a251fdd9fb5aa"
server_key_version: "encryptionkey@0"
I20250109 21:03:10.998133 15800 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250109 21:03:11.000394 15815 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:11.001471 15800 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:03:11.001582 15800 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "367003ed9c524fa093732e9027c566d1"
format_stamp: "Formatted at 2025-01-09 21:03:10 on dist-test-slave-g0zb"
server_key: "b448a3e854a87f66494a1d998436fcad"
server_key_iv: "5768a789165f19a4756a251fdd9fb5aa"
server_key_version: "encryptionkey@0"
I20250109 21:03:11.001689 15800 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:11.017740 15800 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:11.020916 15800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:11.021092 15800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:11.021648 15800 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:11.022467 15800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:11.022514 15800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:11.022557 15800 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:11.022591 15800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:11.031142 15800 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:46813
I20250109 21:03:11.031168 15928 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:46813 every 8 connection(s)
I20250109 21:03:11.032122 15800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:03:11.040655 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 15800
I20250109 21:03:11.040786 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
Jan 09 21:03:11 dist-test-slave-g0zb krb5kdc[15693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456590, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:11.041077 12793 external_mini_cluster.cc:1453] Setting key 9e6289c27e82554c636037b3ae1cd687
I20250109 21:03:11.045521 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:11.033970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:58271 (local address 127.12.126.126:41747)
0109 21:03:11.034170 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:11.034173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:11.035273 (+ 1100us) server_negotiation.cc:407] Connection header received
0109 21:03:11.036083 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:11.036087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:11.036148 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:11.036232 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:11.037588 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:11.038113 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:11.038834 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:11.039022 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:11.041607 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:11.041640 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:11.041647 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:11.041684 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:11.043366 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:11.044030 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:11.044033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:11.044035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:11.044097 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:11.044424 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:11.044427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:11.044429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:11.044598 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:11.044690 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:11.045197 (+ 507us) server_negotiation.cc:299] Negotiation successful
0109 21:03:11.045334 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20250109 21:03:11.046221 15931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:11.034636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:41747 (local address 127.12.126.65:58271)
0109 21:03:11.035107 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:11.035144 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:11.035877 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:11.036403 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:11.036411 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:11.036798 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:11.037388 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:11.037402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:11.038274 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:11.038277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:11.038679 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:11.038687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:11.038956 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:11.039673 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:11.039699 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:11.041403 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:11.043507 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:11.043516 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:11.043533 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:11.043907 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:11.044231 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:11.044233 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:11.044235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:11.044328 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:11.044706 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:11.044713 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:11.044959 (+ 246us) client_negotiation.cc:769] Sending connection context
0109 21:03:11.045148 (+ 189us) client_negotiation.cc:240] Negotiation successful
0109 21:03:11.045446 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":133,"threads_started":1}
I20250109 21:03:11.047614 15929 heartbeater.cc:346] Connected to a master server at 127.12.126.126:41747
I20250109 21:03:11.047863 15929 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:11.048410 15929 heartbeater.cc:510] Master 127.12.126.126:41747 requested a full tablet report, sending...
I20250109 21:03:11.049944 15734 ts_manager.cc:194] Registered new tserver with Master: 367003ed9c524fa093732e9027c566d1 (127.12.126.65:46813)
WARNING: no policy specified for kudu/127.12.126.66@KRBTEST.COM; defaulting to no policy
I20250109 21:03:11.051249 15734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:58271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
I20250109 21:03:11.070873 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:41747
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:45353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:11.178232 15934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:11.178501 15934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:11.178589 15934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:11.181813 15934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:11.181887 15934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:11.181996 15934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:03:11.186415 15934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:45353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:41747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:11.187608 15934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:11.188478 15934 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:11.196758 15942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:11.197582 15939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:11.198397 15940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:11.199112 15934 server_base.cc:1034] running on GCE node
I20250109 21:03:11.199594 15934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:11.200206 15934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:11.201381 15934 hybrid_clock.cc:648] HybridClock initialized: now 1736456591201368 us; error 29 us; skew 500 ppm
Jan 09 21:03:11 dist-test-slave-g0zb krb5kdc[15693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456591, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:11.204402 15934 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:03:11.205402 15934 webserver.cc:458] Webserver started at http://127.12.126.66:42885/ using document root <none> and password file <none>
I20250109 21:03:11.206007 15934 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:11.206079 15934 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:11.206287 15934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:11.208207 15934 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "d468817235f848e0bceea6e468b9a210"
format_stamp: "Formatted at 2025-01-09 21:03:11 on dist-test-slave-g0zb"
server_key: "bb490953c7fd96be77f2a455e4f00b6d"
server_key_iv: "b66cd1c14bdade8e6a0925e69f779b31"
server_key_version: "encryptionkey@0"
I20250109 21:03:11.208691 15934 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "d468817235f848e0bceea6e468b9a210"
format_stamp: "Formatted at 2025-01-09 21:03:11 on dist-test-slave-g0zb"
server_key: "bb490953c7fd96be77f2a455e4f00b6d"
server_key_iv: "b66cd1c14bdade8e6a0925e69f779b31"
server_key_version: "encryptionkey@0"
I20250109 21:03:11.212220 15934 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250109 21:03:11.214579 15949 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:11.215679 15934 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:11.215792 15934 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "d468817235f848e0bceea6e468b9a210"
format_stamp: "Formatted at 2025-01-09 21:03:11 on dist-test-slave-g0zb"
server_key: "bb490953c7fd96be77f2a455e4f00b6d"
server_key_iv: "b66cd1c14bdade8e6a0925e69f779b31"
server_key_version: "encryptionkey@0"
I20250109 21:03:11.215904 15934 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:11.235378 15934 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:11.238488 15934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:11.238659 15934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:11.239266 15934 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:11.240057 15934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:11.240106 15934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:11.240146 15934 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:11.240175 15934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:11.248909 15934 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:40169
I20250109 21:03:11.248935 16062 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:40169 every 8 connection(s)
I20250109 21:03:11.249884 15934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:03:11.256990 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 15934
I20250109 21:03:11.257128 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:03:11.257445 12793 external_mini_cluster.cc:1453] Setting key 91632379edd7bc945dd88e7fceda2147
Jan 09 21:03:11 dist-test-slave-g0zb krb5kdc[15693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456591, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:11.263100 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:11.251735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:51979 (local address 127.12.126.126:41747)
0109 21:03:11.251886 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:11.251891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:11.253345 (+ 1454us) server_negotiation.cc:407] Connection header received
0109 21:03:11.254150 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:11.254154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:11.254212 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:11.254299 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:11.255625 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:11.256143 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:11.256770 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:11.256957 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:11.259404 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:11.259420 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:11.259423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:11.259449 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:11.260996 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:11.261615 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:11.261618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:11.261619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:11.261667 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:11.261971 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:11.261973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:11.261974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:11.262116 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:11.262199 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:11.262872 (+ 673us) server_negotiation.cc:299] Negotiation successful
0109 21:03:11.262981 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250109 21:03:11.264076 16065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:11.252695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:41747 (local address 127.12.126.66:51979)
0109 21:03:11.253188 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:11.253243 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:11.253948 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:11.254466 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:11.254475 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:11.254911 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:11.255486 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:11.255499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:11.256267 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:11.256270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:11.256658 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:11.256665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:11.256906 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:11.257651 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:11.257681 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:11.259251 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:11.261116 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:11.261124 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:11.261140 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:11.261504 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:11.261756 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:11.261762 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:11.261767 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:11.261885 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:11.262219 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:11.262228 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:11.262567 (+ 339us) client_negotiation.cc:769] Sending connection context
0109 21:03:11.262832 (+ 265us) client_negotiation.cc:240] Negotiation successful
0109 21:03:11.263138 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":100,"threads_started":1}
I20250109 21:03:11.265300 16063 heartbeater.cc:346] Connected to a master server at 127.12.126.126:41747
I20250109 21:03:11.265528 16063 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:11.266036 16063 heartbeater.cc:510] Master 127.12.126.126:41747 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.126.67@KRBTEST.COM; defaulting to no policy
I20250109 21:03:11.267113 15734 ts_manager.cc:194] Registered new tserver with Master: d468817235f848e0bceea6e468b9a210 (127.12.126.66:40169)
I20250109 21:03:11.267668 15734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:51979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
I20250109 21:03:11.287182 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:41747
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:45353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:11.394440 16068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:11.394690 16068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:11.394747 16068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:11.397907 16068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:11.397971 16068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:11.398052 16068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:03:11.402365 16068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:45353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:41747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:11.403463 16068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:11.404284 16068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:11.412042 16076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:11.412865 16073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:11.413445 16074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:11.414451 16068 server_base.cc:1034] running on GCE node
I20250109 21:03:11.414880 16068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:11.415475 16068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:11.416652 16068 hybrid_clock.cc:648] HybridClock initialized: now 1736456591416623 us; error 44 us; skew 500 ppm
Jan 09 21:03:11 dist-test-slave-g0zb krb5kdc[15693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456591, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:11.419482 16068 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:03:11.420431 16068 webserver.cc:458] Webserver started at http://127.12.126.67:41063/ using document root <none> and password file <none>
I20250109 21:03:11.420998 16068 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:11.421057 16068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:11.421231 16068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:11.423115 16068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "85069a54747d418a8bf2c4e742cfeefd"
format_stamp: "Formatted at 2025-01-09 21:03:11 on dist-test-slave-g0zb"
server_key: "057b9fb1db99e3b9241107404a7fa4cd"
server_key_iv: "779b697554f4b99f3633a2e2375747a1"
server_key_version: "encryptionkey@0"
I20250109 21:03:11.423611 16068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "85069a54747d418a8bf2c4e742cfeefd"
format_stamp: "Formatted at 2025-01-09 21:03:11 on dist-test-slave-g0zb"
server_key: "057b9fb1db99e3b9241107404a7fa4cd"
server_key_iv: "779b697554f4b99f3633a2e2375747a1"
server_key_version: "encryptionkey@0"
I20250109 21:03:11.426973 16068 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:03:11.429095 16083 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:11.430131 16068 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:11.430241 16068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "85069a54747d418a8bf2c4e742cfeefd"
format_stamp: "Formatted at 2025-01-09 21:03:11 on dist-test-slave-g0zb"
server_key: "057b9fb1db99e3b9241107404a7fa4cd"
server_key_iv: "779b697554f4b99f3633a2e2375747a1"
server_key_version: "encryptionkey@0"
I20250109 21:03:11.430348 16068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:11.468997 16068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:11.472215 16068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:11.472400 16068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:11.472946 16068 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:11.473831 16068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:11.473887 16068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:11.473955 16068 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:11.473984 16068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:11.482703 16068 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:35339
I20250109 21:03:11.482728 16196 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:35339 every 8 connection(s)
I20250109 21:03:11.483685 16068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
Jan 09 21:03:11 dist-test-slave-g0zb krb5kdc[15693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456591, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:11.493547 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 16068
I20250109 21:03:11.493656 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:03:11.493911 12793 external_mini_cluster.cc:1453] Setting key 2f51b59bf1b3c9930e3b2d6a60558ee7
I20250109 21:03:11.495790 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:11.485352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:41133 (local address 127.12.126.126:41747)
0109 21:03:11.485550 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:11.485557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:11.486658 (+ 1101us) server_negotiation.cc:407] Connection header received
0109 21:03:11.487460 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:11.487463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:11.487516 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:11.487595 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:11.488815 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:11.489324 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:11.489881 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:11.490002 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:11.492228 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:11.492253 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:11.492256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:11.492281 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:11.493768 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:11.494280 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:11.494286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:11.494288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:11.494332 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:11.494640 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:11.494646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:11.494651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:11.494880 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:11.494994 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:11.495543 (+ 549us) server_negotiation.cc:299] Negotiation successful
0109 21:03:11.495651 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20250109 21:03:11.496541 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:11.486092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:41747 (local address 127.12.126.67:41133)
0109 21:03:11.486516 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:11.486551 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:11.487270 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:11.487735 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:11.487742 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:11.488125 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:11.488686 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:11.488698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:11.489416 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:11.489419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:11.489797 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:11.489803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:11.489961 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:11.490532 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:11.490554 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:11.492063 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:11.493873 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:11.493878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:11.493891 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:11.494173 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:11.494413 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:11.494416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:11.494418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:11.494537 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:11.494991 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:11.494998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:11.495293 (+ 295us) client_negotiation.cc:769] Sending connection context
0109 21:03:11.495529 (+ 236us) client_negotiation.cc:240] Negotiation successful
0109 21:03:11.495819 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":102,"threads_started":1}
I20250109 21:03:11.497455 16197 heartbeater.cc:346] Connected to a master server at 127.12.126.126:41747
I20250109 21:03:11.497653 16197 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:11.498106 16197 heartbeater.cc:510] Master 127.12.126.126:41747 requested a full tablet report, sending...
I20250109 21:03:11.499032 15734 ts_manager.cc:194] Registered new tserver with Master: 85069a54747d418a8bf2c4e742cfeefd (127.12.126.67:35339)
I20250109 21:03:11.499567 15734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:41133
I20250109 21:03:11.507737 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250109 21:03:11.638283 15777 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:11.634708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.12.126.126:41747)
0109 21:03:11.634881 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:11.634886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:11.635875 (+ 989us) server_negotiation.cc:407] Connection header received
0109 21:03:11.636775 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:11.636779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:11.636835 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:11.636921 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:11.638144 (+ 1223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50548: BlockingRecv error: recv got EOF from 127.0.0.1:50548 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20250109 21:03:11.639626 15777 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:11.638891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.12.126.126:41747)
0109 21:03:11.639014 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:11.639017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:11.639029 (+ 12us) server_negotiation.cc:407] Connection header received
0109 21:03:11.639120 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:11.639122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:11.639157 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:11.639223 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:11.639544 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50550: BlockingRecv error: recv got EOF from 127.0.0.1:50550 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20250109 21:03:11.640511 15777 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:11.639893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.12.126.126:41747)
0109 21:03:11.639992 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:11.639995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:11.640006 (+ 11us) server_negotiation.cc:407] Connection header received
0109 21:03:11.640079 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:11.640081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:11.640111 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:11.640171 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:11.640428 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50564: BlockingRecv error: recv got EOF from 127.0.0.1:50564 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20250109 21:03:11.644279 15777 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:11.643475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.12.126.126:41747)
0109 21:03:11.643582 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:11.643586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:11.643611 (+ 25us) server_negotiation.cc:407] Connection header received
0109 21:03:11.643639 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:11.643641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:11.643673 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:11.643731 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:11.644159 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50568: BlockingRecv error: recv got EOF from 127.0.0.1:50568 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20250109 21:03:11.652016 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 15800
I20250109 21:03:11.657691 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 15934
I20250109 21:03:11.663192 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 16068
I20250109 21:03:11.668386 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 15706
2025-01-09T21:03:11Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (2270 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:03:11 dist-test-slave-g0zb krb5kdc[16220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:03:11 dist-test-slave-g0zb krb5kdc[16220](info): set up 2 sockets
Jan 09 21:03:11 dist-test-slave-g0zb krb5kdc[16220](info): commencing operation
krb5kdc: starting...
W20250109 21:03:12.699652 12793 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:03:12 dist-test-slave-g0zb krb5kdc[16220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:03:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:03:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:03:12.817489 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:35205
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:35205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:12.927832 16233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:12.928107 16233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:12.928194 16233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:12.932874 16233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:12.932976 16233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:12.933012 16233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:12.933048 16233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:12.933081 16233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:03:12.938393 16233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:35205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:35205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:12.939591 16233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:12.940438 16233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:12.947705 16238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:12.948334 16239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:12.949373 16241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:12.950373 16233 server_base.cc:1034] running on GCE node
I20250109 21:03:12.950987 16233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:12.952035 16233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:12.953214 16233 hybrid_clock.cc:648] HybridClock initialized: now 1736456592953173 us; error 41 us; skew 500 ppm
Jan 09 21:03:12 dist-test-slave-g0zb krb5kdc[16220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456592, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:12.955996 16233 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:03:12.956991 16233 webserver.cc:458] Webserver started at http://127.12.126.126:37465/ using document root <none> and password file <none>
I20250109 21:03:12.957567 16233 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:12.957630 16233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:12.957828 16233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:12.959735 16233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "b1932ab392054e16a56ebf754231e817"
format_stamp: "Formatted at 2025-01-09 21:03:12 on dist-test-slave-g0zb"
server_key: "a800a635280700cdaad149cabee81cdc"
server_key_iv: "5b1bf45b70a9dc07fdb0acd5a54a1d63"
server_key_version: "encryptionkey@0"
I20250109 21:03:12.960218 16233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "b1932ab392054e16a56ebf754231e817"
format_stamp: "Formatted at 2025-01-09 21:03:12 on dist-test-slave-g0zb"
server_key: "a800a635280700cdaad149cabee81cdc"
server_key_iv: "5b1bf45b70a9dc07fdb0acd5a54a1d63"
server_key_version: "encryptionkey@0"
I20250109 21:03:12.963588 16233 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:03:12.965776 16248 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:12.966912 16233 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:12.967044 16233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "b1932ab392054e16a56ebf754231e817"
format_stamp: "Formatted at 2025-01-09 21:03:12 on dist-test-slave-g0zb"
server_key: "a800a635280700cdaad149cabee81cdc"
server_key_iv: "5b1bf45b70a9dc07fdb0acd5a54a1d63"
server_key_version: "encryptionkey@0"
I20250109 21:03:12.967135 16233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:12.980993 16233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:12.983897 16233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:12.984059 16233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:12.990696 16300 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:35205 every 8 connection(s)
I20250109 21:03:12.990693 16233 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:35205
I20250109 21:03:12.991750 16233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:03:12.993206 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 16233
I20250109 21:03:12.993324 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:03:12.993567 12793 external_mini_cluster.cc:1453] Setting key 822a8c1f022d2ae780fb63e094c236f6
I20250109 21:03:12.994850 16301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:13.000994 16301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817: Bootstrap starting.
Jan 09 21:03:12 dist-test-slave-g0zb krb5kdc[16220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:13.003544 16301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:13.004391 16301 log.cc:826] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:13.006160 16301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817: No bootstrap required, opened a new log
I20250109 21:03:13.006755 16304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:12.994622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.12.126.126:35205)
0109 21:03:12.995026 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:12.995035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:12.995069 (+ 34us) server_negotiation.cc:407] Connection header received
0109 21:03:12.995653 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:12.995675 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:12.995958 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:12.996268 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:12.997421 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:12.998220 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:12.999089 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:12.999359 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:13.001821 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:13.001887 (+ 66us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:13.001904 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:13.001950 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:13.004153 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:13.004595 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:13.004600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:13.004604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:13.004677 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:13.004889 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:13.004893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:13.004895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:13.005235 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:13.005434 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:13.005705 (+ 271us) server_negotiation.cc:299] Negotiation successful
0109 21:03:13.005957 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":134,"threads_started":1}
I20250109 21:03:13.008853 16301 raft_consensus.cc:357] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1932ab392054e16a56ebf754231e817" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35205 } }
I20250109 21:03:13.009039 16301 raft_consensus.cc:383] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:13.009109 16301 raft_consensus.cc:738] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1932ab392054e16a56ebf754231e817, State: Initialized, Role: FOLLOWER
I20250109 21:03:13.009331 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:03:13.009544 16301 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1932ab392054e16a56ebf754231e817" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35205 } }
I20250109 21:03:13.009670 16301 raft_consensus.cc:397] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:03:13.009725 16301 raft_consensus.cc:491] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:03:13.009824 16301 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:13.010639 16301 raft_consensus.cc:513] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1932ab392054e16a56ebf754231e817" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35205 } }
I20250109 21:03:13.011030 16301 leader_election.cc:304] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1932ab392054e16a56ebf754231e817; no voters:
I20250109 21:03:13.011375 16301 leader_election.cc:290] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:03:13.011442 16307 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:13.011658 16307 raft_consensus.cc:695] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817 [term 1 LEADER]: Becoming Leader. State: Replica: b1932ab392054e16a56ebf754231e817, State: Running, Role: LEADER
I20250109 21:03:13.011996 16307 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1932ab392054e16a56ebf754231e817" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35205 } }
I20250109 21:03:13.012614 16301 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:13.013547 16308 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1932ab392054e16a56ebf754231e817" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1932ab392054e16a56ebf754231e817" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35205 } } }
I20250109 21:03:13.013546 16309 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1932ab392054e16a56ebf754231e817. Latest consensus state: current_term: 1 leader_uuid: "b1932ab392054e16a56ebf754231e817" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1932ab392054e16a56ebf754231e817" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35205 } } }
I20250109 21:03:13.013686 16308 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:13.013686 16309 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:13.014163 16316 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:03:13.017649 16316 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:03:13.023967 16316 catalog_manager.cc:1348] Generated new cluster ID: 1ee953b145ac4af6b9c554dcf0904d51
I20250109 21:03:13.024102 16316 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:03:13.026125 16304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:13.021765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.12.126.126:35205)
0109 21:03:13.022081 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:13.022084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:13.022096 (+ 12us) server_negotiation.cc:407] Connection header received
0109 21:03:13.022128 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:13.022130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:13.022185 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:13.022269 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:13.022926 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:13.023392 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:13.024061 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:13.024257 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:13.025183 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:13.025199 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:13.025203 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:03:13.025677 (+ 474us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:13.025716 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:03:13.025737 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:13.025835 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:13.025904 (+ 69us) server_negotiation.cc:299] Negotiation successful
0109 21:03:13.025976 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258}
I20250109 21:03:13.034925 16316 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:03:13.035936 16316 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:03:13.051275 16316 catalog_manager.cc:5899] T 00000000000000000000000000000000 P b1932ab392054e16a56ebf754231e817: Generated new TSK 0
I20250109 21:03:13.052001 16316 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:03:13.058344 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 16233
2025-01-09T21:03:13Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1387 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:03:13 dist-test-slave-g0zb krb5kdc[16334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:03:13 dist-test-slave-g0zb krb5kdc[16334](info): set up 2 sockets
Jan 09 21:03:13 dist-test-slave-g0zb krb5kdc[16334](info): commencing operation
krb5kdc: starting...
W20250109 21:03:14.107486 12793 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:03:14 dist-test-slave-g0zb krb5kdc[16334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:03:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:03:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:03:14.226006 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:33303
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:33303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:14.337791 16347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:14.338055 16347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:14.338150 16347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:14.341394 16347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:14.341465 16347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:14.341503 16347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:14.341547 16347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:14.341589 16347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:03:14.346029 16347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:33303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:33303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:14.347201 16347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:14.348078 16347 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:14.355006 16352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:14.355777 16353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:14.356693 16355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:14.357445 16347 server_base.cc:1034] running on GCE node
I20250109 21:03:14.357966 16347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:14.358954 16347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:14.360101 16347 hybrid_clock.cc:648] HybridClock initialized: now 1736456594360082 us; error 35 us; skew 500 ppm
Jan 09 21:03:14 dist-test-slave-g0zb krb5kdc[16334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456594, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:14.362943 16347 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:03:14.363895 16347 webserver.cc:458] Webserver started at http://127.12.126.126:35593/ using document root <none> and password file <none>
I20250109 21:03:14.364437 16347 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:14.364498 16347 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:14.364681 16347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:14.366467 16347 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "4474b222aaab41599ee876a24ede30ad"
format_stamp: "Formatted at 2025-01-09 21:03:14 on dist-test-slave-g0zb"
server_key: "8b2d0e61299706d4ed60cd2da1705cda"
server_key_iv: "72b7f43835605929b103a39635eb0199"
server_key_version: "encryptionkey@0"
I20250109 21:03:14.366974 16347 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "4474b222aaab41599ee876a24ede30ad"
format_stamp: "Formatted at 2025-01-09 21:03:14 on dist-test-slave-g0zb"
server_key: "8b2d0e61299706d4ed60cd2da1705cda"
server_key_iv: "72b7f43835605929b103a39635eb0199"
server_key_version: "encryptionkey@0"
I20250109 21:03:14.370393 16347 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:03:14.372712 16362 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:14.373801 16347 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:14.373911 16347 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "4474b222aaab41599ee876a24ede30ad"
format_stamp: "Formatted at 2025-01-09 21:03:14 on dist-test-slave-g0zb"
server_key: "8b2d0e61299706d4ed60cd2da1705cda"
server_key_iv: "72b7f43835605929b103a39635eb0199"
server_key_version: "encryptionkey@0"
I20250109 21:03:14.374002 16347 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:14.405006 16347 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:14.407984 16347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:14.408180 16347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:14.415203 16347 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:33303
I20250109 21:03:14.415207 16414 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:33303 every 8 connection(s)
I20250109 21:03:14.416252 16347 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:03:14.419252 16415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:14.422303 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 16347
I20250109 21:03:14.422443 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:03:14.422706 12793 external_mini_cluster.cc:1453] Setting key a107244b03bd2cfec74ae7078b5a76f0
I20250109 21:03:14.424803 16415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad: Bootstrap starting.
I20250109 21:03:14.427026 16415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:14.427762 16415 log.cc:826] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:14.429541 16415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad: No bootstrap required, opened a new log
Jan 09 21:03:14 dist-test-slave-g0zb krb5kdc[16334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:14.433352 16415 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4474b222aaab41599ee876a24ede30ad" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33303 } }
I20250109 21:03:14.433560 16415 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:14.433679 16415 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4474b222aaab41599ee876a24ede30ad, State: Initialized, Role: FOLLOWER
I20250109 21:03:14.434190 16415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4474b222aaab41599ee876a24ede30ad" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33303 } }
I20250109 21:03:14.434352 16415 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:03:14.434418 16415 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:03:14.434537 16415 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:14.435813 16415 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4474b222aaab41599ee876a24ede30ad" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33303 } }
I20250109 21:03:14.436214 16415 leader_election.cc:304] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4474b222aaab41599ee876a24ede30ad; no voters:
I20250109 21:03:14.436538 16415 leader_election.cc:290] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:03:14.436543 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:14.424053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.12.126.126:33303)
0109 21:03:14.424486 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:14.424496 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:14.424533 (+ 37us) server_negotiation.cc:407] Connection header received
0109 21:03:14.425167 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:14.425191 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:14.425508 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:14.425855 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:14.426774 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:14.427667 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:14.428337 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:14.428626 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:14.431067 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:14.431086 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:14.431098 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:14.431131 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:14.433896 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:14.434323 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:14.434331 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:14.434339 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:14.434447 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:14.434698 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:14.434702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:14.434704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:14.435095 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:14.435328 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:14.435552 (+ 224us) server_negotiation.cc:299] Negotiation successful
0109 21:03:14.435846 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":131,"threads_started":1}
I20250109 21:03:14.436673 16420 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:14.437072 16420 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad [term 1 LEADER]: Becoming Leader. State: Replica: 4474b222aaab41599ee876a24ede30ad, State: Running, Role: LEADER
I20250109 21:03:14.437357 16420 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4474b222aaab41599ee876a24ede30ad" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33303 } }
I20250109 21:03:14.437896 16415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:14.438390 16420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4474b222aaab41599ee876a24ede30ad. Latest consensus state: current_term: 1 leader_uuid: "4474b222aaab41599ee876a24ede30ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4474b222aaab41599ee876a24ede30ad" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33303 } } }
I20250109 21:03:14.438513 16420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:14.438774 16421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4474b222aaab41599ee876a24ede30ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4474b222aaab41599ee876a24ede30ad" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33303 } } }
I20250109 21:03:14.438944 16421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad [sys.catalog]: This master's current role is: LEADER
W20250109 21:03:14.443323 16435 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:03:14.443398 16435 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 21:03:14.443493 16424 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:03:14.444240 16424 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:03:14.449926 16424 catalog_manager.cc:1348] Generated new cluster ID: 19d0e5020e3a4de78b4b1c5e410ab921
I20250109 21:03:14.450012 16424 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:03:14.464156 16424 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:03:14.465456 16424 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:03:14.478328 16424 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 4474b222aaab41599ee876a24ede30ad: Generated new TSK 0
I20250109 21:03:14.479247 16424 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
W20250109 21:03:15.927251 12793 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.12.126.65: real 1.441s user 0.001s sys 0.002s
I20250109 21:03:15.927765 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:33303
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:16.035725 16441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:16.035974 16441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:16.036055 16441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:16.039149 16441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:16.039211 16441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:16.039311 16441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:03:16.043485 16441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:33303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:16.044561 16441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:16.045418 16441 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:16.054126 16447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:16.054677 16446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:16.055580 16449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:16.056362 16441 server_base.cc:1034] running on GCE node
I20250109 21:03:16.056735 16441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:16.057313 16441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:16.058456 16441 hybrid_clock.cc:648] HybridClock initialized: now 1736456596058441 us; error 29 us; skew 500 ppm
Jan 09 21:03:16 dist-test-slave-g0zb krb5kdc[16334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456596, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:16.066548 16441 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:03:16.067499 16441 webserver.cc:458] Webserver started at http://127.12.126.65:38587/ using document root <none> and password file <none>
I20250109 21:03:16.068087 16441 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:16.068143 16441 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:16.068341 16441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:16.070159 16441 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "e5ec4648026a47c096fa870f8f6b4329"
format_stamp: "Formatted at 2025-01-09 21:03:16 on dist-test-slave-g0zb"
server_key: "a2f7d7ee7aabe994c6cd5dcd60fa2c0e"
server_key_iv: "cb5d447967504f36f1a1315d248b1a19"
server_key_version: "encryptionkey@0"
I20250109 21:03:16.070638 16441 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "e5ec4648026a47c096fa870f8f6b4329"
format_stamp: "Formatted at 2025-01-09 21:03:16 on dist-test-slave-g0zb"
server_key: "a2f7d7ee7aabe994c6cd5dcd60fa2c0e"
server_key_iv: "cb5d447967504f36f1a1315d248b1a19"
server_key_version: "encryptionkey@0"
I20250109 21:03:16.074193 16441 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250109 21:03:16.076481 16456 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:16.077518 16441 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250109 21:03:16.077612 16441 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "e5ec4648026a47c096fa870f8f6b4329"
format_stamp: "Formatted at 2025-01-09 21:03:16 on dist-test-slave-g0zb"
server_key: "a2f7d7ee7aabe994c6cd5dcd60fa2c0e"
server_key_iv: "cb5d447967504f36f1a1315d248b1a19"
server_key_version: "encryptionkey@0"
I20250109 21:03:16.077703 16441 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:16.092348 16441 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:16.095597 16441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:16.095770 16441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:16.096321 16441 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:16.097090 16441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:16.097136 16441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:16.097178 16441 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:16.097222 16441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:16.105801 16441 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:43641
I20250109 21:03:16.105820 16569 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:43641 every 8 connection(s)
I20250109 21:03:16.106770 16441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:03:16.113202 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 16441
I20250109 21:03:16.113332 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:03:16.113586 12793 external_mini_cluster.cc:1453] Setting key 88ddfdc45081c3beece777e74ad00624
Jan 09 21:03:16 dist-test-slave-g0zb krb5kdc[16334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456596, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:16.120517 16572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:16.108497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:43087 (local address 127.12.126.126:33303)
0109 21:03:16.108778 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:16.108782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:16.109782 (+ 1000us) server_negotiation.cc:407] Connection header received
0109 21:03:16.110595 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:16.110599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:16.110665 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:16.110755 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:16.112092 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:16.112627 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:16.113285 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:16.113468 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:16.116310 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:16.116333 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:16.116340 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:16.116377 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:16.117982 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:16.118623 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:16.118629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:16.118633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:16.118699 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:16.119045 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:16.119052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:16.119056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:16.119270 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:16.119377 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:16.120077 (+ 700us) server_negotiation.cc:299] Negotiation successful
0109 21:03:16.120224 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":104,"threads_started":1}
I20250109 21:03:16.121294 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:16.109229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33303 (local address 127.12.126.65:43087)
0109 21:03:16.109647 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:16.109684 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:16.110377 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:16.110950 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:16.110958 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:16.111338 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:16.111930 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:16.111944 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:16.112750 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:16.112753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:16.113160 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:16.113168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:16.113381 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:16.114090 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:16.114123 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:16.116123 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:16.118099 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:16.118112 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:16.118128 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:16.118512 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:16.118820 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:16.118823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:16.118825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:16.118954 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:16.119398 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:16.119408 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:16.119767 (+ 359us) client_negotiation.cc:769] Sending connection context
0109 21:03:16.120025 (+ 258us) client_negotiation.cc:240] Negotiation successful
0109 21:03:16.120375 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
I20250109 21:03:16.122529 16570 heartbeater.cc:346] Connected to a master server at 127.12.126.126:33303
I20250109 21:03:16.122750 16570 heartbeater.cc:463] Registering TS with master...
WARNING: no policy specified for kudu/127.12.126.66@KRBTEST.COM; defaulting to no policy
I20250109 21:03:16.123299 16570 heartbeater.cc:510] Master 127.12.126.126:33303 requested a full tablet report, sending...
I20250109 21:03:16.124743 16379 ts_manager.cc:194] Registered new tserver with Master: e5ec4648026a47c096fa870f8f6b4329 (127.12.126.65:43641)
I20250109 21:03:16.125914 16379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:43087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.66@KRBTEST.COM" created.
I20250109 21:03:17.128993 16570 heartbeater.cc:502] Master 127.12.126.126:33303 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
W20250109 21:03:17.595052 12793 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.12.126.66: real 1.480s user 0.001s sys 0.002s
I20250109 21:03:17.595652 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:33303
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:17.703748 16576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:17.703989 16576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:17.704062 16576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:17.707181 16576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:17.707250 16576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:17.707353 16576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:03:17.711567 16576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:33303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:17.712638 16576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:17.713478 16576 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:17.722338 16581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:17.723167 16582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:17.723946 16584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:17.725224 16576 server_base.cc:1034] running on GCE node
I20250109 21:03:17.725656 16576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:17.726248 16576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:17.727394 16576 hybrid_clock.cc:648] HybridClock initialized: now 1736456597727381 us; error 32 us; skew 500 ppm
Jan 09 21:03:17 dist-test-slave-g0zb krb5kdc[16334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456597, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:17.730041 16576 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:03:17.731101 16576 webserver.cc:458] Webserver started at http://127.12.126.66:46737/ using document root <none> and password file <none>
I20250109 21:03:17.731644 16576 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:17.731695 16576 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:17.731855 16576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:17.733606 16576 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "c337b3ad35d547f690aff7a91f54f5b4"
format_stamp: "Formatted at 2025-01-09 21:03:17 on dist-test-slave-g0zb"
server_key: "05dde365b0c43431efb70faa682e85fa"
server_key_iv: "69b68e78137cfbdc0b56f53309c4b0e0"
server_key_version: "encryptionkey@0"
I20250109 21:03:17.734043 16576 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "c337b3ad35d547f690aff7a91f54f5b4"
format_stamp: "Formatted at 2025-01-09 21:03:17 on dist-test-slave-g0zb"
server_key: "05dde365b0c43431efb70faa682e85fa"
server_key_iv: "69b68e78137cfbdc0b56f53309c4b0e0"
server_key_version: "encryptionkey@0"
I20250109 21:03:17.737373 16576 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:03:17.739486 16591 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:17.740506 16576 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:17.740617 16576 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "c337b3ad35d547f690aff7a91f54f5b4"
format_stamp: "Formatted at 2025-01-09 21:03:17 on dist-test-slave-g0zb"
server_key: "05dde365b0c43431efb70faa682e85fa"
server_key_iv: "69b68e78137cfbdc0b56f53309c4b0e0"
server_key_version: "encryptionkey@0"
I20250109 21:03:17.740707 16576 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:17.760769 16576 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:17.763602 16576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:17.763744 16576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:17.764215 16576 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:17.764986 16576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:17.765033 16576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:17.765077 16576 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:17.765108 16576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:17.773581 16576 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:41411
I20250109 21:03:17.773608 16704 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:41411 every 8 connection(s)
I20250109 21:03:17.774523 16576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:03:17.781386 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 16576
I20250109 21:03:17.781514 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:03:17.782755 12793 external_mini_cluster.cc:1453] Setting key 2ff7c94f9aee1e1bc59d25804204afd0
Jan 09 21:03:17 dist-test-slave-g0zb krb5kdc[16334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456597, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:17.788270 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:17.776270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:56339 (local address 127.12.126.126:33303)
0109 21:03:17.776498 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:17.776501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:17.777818 (+ 1317us) server_negotiation.cc:407] Connection header received
0109 21:03:17.778836 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:17.778841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:17.778910 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:17.779003 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:17.780372 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:17.780913 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:17.781714 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:17.781861 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:17.784615 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:17.784631 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:17.784633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:17.784657 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:17.786343 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:17.786902 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:17.786906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:17.786908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:17.786964 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:17.787231 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:17.787235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:17.787237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:17.787392 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:17.787488 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:17.787962 (+ 474us) server_negotiation.cc:299] Negotiation successful
0109 21:03:17.788093 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":65,"threads_started":1}
I20250109 21:03:17.788900 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:17.777166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33303 (local address 127.12.126.66:56339)
0109 21:03:17.777669 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:17.777705 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:17.778574 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:17.779228 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:17.779237 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:17.779626 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:17.780217 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:17.780235 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:17.781040 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:17.781043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:17.781599 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:17.781612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:17.781822 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:17.782447 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:17.782477 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:17.784398 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:17.786455 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:17.786462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:17.786475 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:17.786765 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:17.787044 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:17.787047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:17.787049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:17.787141 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:17.787497 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:17.787506 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:17.787746 (+ 240us) client_negotiation.cc:769] Sending connection context
0109 21:03:17.787918 (+ 172us) client_negotiation.cc:240] Negotiation successful
0109 21:03:17.788206 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":142,"threads_started":1}
I20250109 21:03:17.790144 16705 heartbeater.cc:346] Connected to a master server at 127.12.126.126:33303
I20250109 21:03:17.790359 16705 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:17.790895 16705 heartbeater.cc:510] Master 127.12.126.126:33303 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.126.67@KRBTEST.COM; defaulting to no policy
I20250109 21:03:17.791852 16379 ts_manager.cc:194] Registered new tserver with Master: c337b3ad35d547f690aff7a91f54f5b4 (127.12.126.66:41411)
I20250109 21:03:17.792387 16379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:56339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
W20250109 21:03:18.203193 12793 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.12.126.67: real 0.419s user 0.000s sys 0.004s
I20250109 21:03:18.203759 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:33303
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:18.310647 16711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:18.310940 16711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:18.311008 16711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:18.314210 16711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:18.314281 16711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:18.314381 16711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:03:18.318783 16711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:33303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:18.319902 16711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:18.320750 16711 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:18.329442 16716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:18.330102 16719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:18.331079 16717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:18.331851 16711 server_base.cc:1034] running on GCE node
I20250109 21:03:18.332232 16711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:18.332798 16711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:18.333945 16711 hybrid_clock.cc:648] HybridClock initialized: now 1736456598333928 us; error 30 us; skew 500 ppm
Jan 09 21:03:18 dist-test-slave-g0zb krb5kdc[16334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456598, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:18.336658 16711 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:03:18.337525 16711 webserver.cc:458] Webserver started at http://127.12.126.67:46181/ using document root <none> and password file <none>
I20250109 21:03:18.338056 16711 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:18.338106 16711 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:18.338255 16711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:18.340168 16711 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "ba439f85285f49f1b0036564d3736b2d"
format_stamp: "Formatted at 2025-01-09 21:03:18 on dist-test-slave-g0zb"
server_key: "9db4e1faeeaf9f4a43e923e8d854ae7c"
server_key_iv: "0a98821f5e35af13df4a20eec8f38489"
server_key_version: "encryptionkey@0"
I20250109 21:03:18.340618 16711 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "ba439f85285f49f1b0036564d3736b2d"
format_stamp: "Formatted at 2025-01-09 21:03:18 on dist-test-slave-g0zb"
server_key: "9db4e1faeeaf9f4a43e923e8d854ae7c"
server_key_iv: "0a98821f5e35af13df4a20eec8f38489"
server_key_version: "encryptionkey@0"
I20250109 21:03:18.344005 16711 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:03:18.346117 16726 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:18.347083 16711 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:03:18.347182 16711 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "ba439f85285f49f1b0036564d3736b2d"
format_stamp: "Formatted at 2025-01-09 21:03:18 on dist-test-slave-g0zb"
server_key: "9db4e1faeeaf9f4a43e923e8d854ae7c"
server_key_iv: "0a98821f5e35af13df4a20eec8f38489"
server_key_version: "encryptionkey@0"
I20250109 21:03:18.347272 16711 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:18.363819 16711 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:18.366731 16711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:18.366930 16711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:18.367441 16711 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:18.368203 16711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:18.368250 16711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:18.368291 16711 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:18.368321 16711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:18.376420 16711 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:40903
I20250109 21:03:18.376518 16839 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:40903 every 8 connection(s)
I20250109 21:03:18.377374 16711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:03:18.379566 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 16711
I20250109 21:03:18.379693 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:03:18.379953 12793 external_mini_cluster.cc:1453] Setting key b79ecbd0c485b56069c309c2f27e8456
Jan 09 21:03:18 dist-test-slave-g0zb krb5kdc[16334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456598, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:18.389766 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:18.379092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:49233 (local address 127.12.126.126:33303)
0109 21:03:18.379355 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:18.379358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:18.380584 (+ 1226us) server_negotiation.cc:407] Connection header received
0109 21:03:18.381456 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:18.381460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:18.381520 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:18.381607 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:18.383093 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:18.383625 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:18.384248 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:18.384431 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:18.386522 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:18.386540 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:18.386542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:18.386568 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:18.388106 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:18.388560 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:18.388563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:18.388564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:18.388615 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:18.388845 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:18.388847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:18.388848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:18.388991 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:18.389061 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:18.389516 (+ 455us) server_negotiation.cc:299] Negotiation successful
0109 21:03:18.389626 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
I20250109 21:03:18.390345 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:18.379975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33303 (local address 127.12.126.67:49233)
0109 21:03:18.380453 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:18.380504 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:18.381250 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:18.381769 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:18.381776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:18.382297 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:18.382963 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:18.382978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:18.383741 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:18.383744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:18.384136 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:18.384143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:18.384354 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:18.384966 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:18.384991 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:18.386364 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:18.388221 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:18.388225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:18.388235 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:18.388463 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:18.388688 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:18.388690 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:18.388691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:18.388767 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:18.389068 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:18.389073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:18.389295 (+ 222us) client_negotiation.cc:769] Sending connection context
0109 21:03:18.389451 (+ 156us) client_negotiation.cc:240] Negotiation successful
0109 21:03:18.389678 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":119,"threads_started":1}
I20250109 21:03:18.391376 16840 heartbeater.cc:346] Connected to a master server at 127.12.126.126:33303
I20250109 21:03:18.391599 16840 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:18.392091 16840 heartbeater.cc:510] Master 127.12.126.126:33303 requested a full tablet report, sending...
I20250109 21:03:18.393110 16378 ts_manager.cc:194] Registered new tserver with Master: ba439f85285f49f1b0036564d3736b2d (127.12.126.67:40903)
I20250109 21:03:18.393781 16378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:49233
I20250109 21:03:18.393844 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
Jan 09 21:03:18 dist-test-slave-g0zb krb5kdc[16334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456598, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jan 09 21:03:18 dist-test-slave-g0zb krb5kdc[16334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456598, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:18.417233 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:18.409500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41994 (local address 127.12.126.126:33303)
0109 21:03:18.409635 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:18.409639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:18.409812 (+ 173us) server_negotiation.cc:407] Connection header received
0109 21:03:18.409933 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:18.409936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:18.409983 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:18.410065 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:18.410878 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:18.411388 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:18.411997 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:18.412129 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:18.414301 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:18.414321 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:18.414323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:18.414350 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:18.415831 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:18.416188 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:18.416194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:18.416197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:18.416245 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:18.416445 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:18.416448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:18.416450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:18.416623 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:18.416744 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:18.416954 (+ 210us) server_negotiation.cc:299] Negotiation successful
0109 21:03:18.417079 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20250109 21:03:18.417575 16378 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41994
I20250109 21:03:18.418396 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 16441
I20250109 21:03:18.424584 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 16576
I20250109 21:03:18.430132 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 16711
I20250109 21:03:18.435603 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 16347
2025-01-09T21:03:18Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (5379 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:03:18 dist-test-slave-g0zb krb5kdc[16853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:03:18 dist-test-slave-g0zb krb5kdc[16853](info): set up 2 sockets
Jan 09 21:03:18 dist-test-slave-g0zb krb5kdc[16853](info): commencing operation
krb5kdc: starting...
W20250109 21:03:19.471525 12793 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:03:19 dist-test-slave-g0zb krb5kdc[16853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:03:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:03:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:03:19.584100 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:41975
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:45547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:41975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:19.687163 16866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:19.687407 16866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:19.687461 16866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:19.690560 16866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:19.690620 16866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:19.690642 16866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:19.690667 16866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:19.690690 16866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:03:19.694851 16866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:45547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:41975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:41975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:19.695788 16866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:19.696571 16866 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:19.703796 16872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:19.704741 16871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:19.705509 16874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:19.706357 16866 server_base.cc:1034] running on GCE node
I20250109 21:03:19.706862 16866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:19.707851 16866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:19.709026 16866 hybrid_clock.cc:648] HybridClock initialized: now 1736456599709002 us; error 36 us; skew 500 ppm
Jan 09 21:03:19 dist-test-slave-g0zb krb5kdc[16853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456599, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:19.711622 16866 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:03:19.712482 16866 webserver.cc:458] Webserver started at http://127.12.126.126:37893/ using document root <none> and password file <none>
I20250109 21:03:19.713002 16866 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:19.713060 16866 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:19.713272 16866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:19.715097 16866 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "f3d56271c8e04c13b508b86323147265"
format_stamp: "Formatted at 2025-01-09 21:03:19 on dist-test-slave-g0zb"
server_key: "f4f92cc9b9d680adc52b4139a891a880"
server_key_iv: "bf3bc269ad501e9491fad2481d275582"
server_key_version: "encryptionkey@0"
I20250109 21:03:19.715586 16866 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "f3d56271c8e04c13b508b86323147265"
format_stamp: "Formatted at 2025-01-09 21:03:19 on dist-test-slave-g0zb"
server_key: "f4f92cc9b9d680adc52b4139a891a880"
server_key_iv: "bf3bc269ad501e9491fad2481d275582"
server_key_version: "encryptionkey@0"
I20250109 21:03:19.718926 16866 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:03:19.721109 16881 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:19.722087 16866 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:19.722225 16866 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "f3d56271c8e04c13b508b86323147265"
format_stamp: "Formatted at 2025-01-09 21:03:19 on dist-test-slave-g0zb"
server_key: "f4f92cc9b9d680adc52b4139a891a880"
server_key_iv: "bf3bc269ad501e9491fad2481d275582"
server_key_version: "encryptionkey@0"
I20250109 21:03:19.722311 16866 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:19.748798 16866 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:19.751457 16866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:19.751616 16866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:19.757990 16866 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:41975
I20250109 21:03:19.758204 16933 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:41975 every 8 connection(s)
I20250109 21:03:19.758999 16866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:03:19.761567 16934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:19.766346 16934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265: Bootstrap starting.
I20250109 21:03:19.768528 16934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:19.769179 16934 log.cc:826] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:19.769197 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 16866
I20250109 21:03:19.769322 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:03:19.769589 12793 external_mini_cluster.cc:1453] Setting key ded306e393fcaa87ef016b1382bb82aa
I20250109 21:03:19.771193 16934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265: No bootstrap required, opened a new log
I20250109 21:03:19.773995 16934 raft_consensus.cc:357] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d56271c8e04c13b508b86323147265" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41975 } }
I20250109 21:03:19.774173 16934 raft_consensus.cc:383] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:19.774247 16934 raft_consensus.cc:738] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3d56271c8e04c13b508b86323147265, State: Initialized, Role: FOLLOWER
I20250109 21:03:19.774641 16934 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d56271c8e04c13b508b86323147265" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41975 } }
I20250109 21:03:19.774763 16934 raft_consensus.cc:397] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:03:19.774873 16934 raft_consensus.cc:491] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:03:19.775024 16934 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:19.775794 16934 raft_consensus.cc:513] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d56271c8e04c13b508b86323147265" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41975 } }
I20250109 21:03:19.776125 16934 leader_election.cc:304] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3d56271c8e04c13b508b86323147265; no voters:
I20250109 21:03:19.776412 16934 leader_election.cc:290] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:03:19.776540 16939 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:19.776734 16939 raft_consensus.cc:695] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265 [term 1 LEADER]: Becoming Leader. State: Replica: f3d56271c8e04c13b508b86323147265, State: Running, Role: LEADER
Jan 09 21:03:19 dist-test-slave-g0zb krb5kdc[16853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:19.777113 16939 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d56271c8e04c13b508b86323147265" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41975 } }
I20250109 21:03:19.777387 16934 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:19.778745 16940 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3d56271c8e04c13b508b86323147265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d56271c8e04c13b508b86323147265" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41975 } } }
I20250109 21:03:19.778898 16940 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:19.779098 16940 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3d56271c8e04c13b508b86323147265. Latest consensus state: current_term: 1 leader_uuid: "f3d56271c8e04c13b508b86323147265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d56271c8e04c13b508b86323147265" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41975 } } }
I20250109 21:03:19.779188 16940 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:19.779407 16949 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:03:19.782440 16949 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:03:19.782440 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:19.770671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55018 (local address 127.12.126.126:41975)
0109 21:03:19.771057 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:19.771066 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:19.771097 (+ 31us) server_negotiation.cc:407] Connection header received
0109 21:03:19.771668 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:19.771686 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:19.771947 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:19.772268 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:19.773172 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:19.773925 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:19.774610 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:19.774941 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:19.777475 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:19.777505 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:19.777519 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:19.777555 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:19.780078 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:19.780531 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:19.780537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:19.780542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:19.780627 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:19.780834 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:19.780838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:19.780839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:19.781169 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:19.781375 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:19.781604 (+ 229us) server_negotiation.cc:299] Negotiation successful
0109 21:03:19.781882 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":134,"threads_started":1}
I20250109 21:03:19.787531 16949 catalog_manager.cc:1348] Generated new cluster ID: 74bb608a0aa64442bdf013666b5fdbd5
I20250109 21:03:19.787638 16949 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:03:19.796619 16949 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:03:19.797495 16949 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:03:19.806196 16949 catalog_manager.cc:5899] T 00000000000000000000000000000000 P f3d56271c8e04c13b508b86323147265: Generated new TSK 0
I20250109 21:03:19.806865 16949 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
I20250109 21:03:19.838152 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:41975
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:45547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:19.965257 16960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:19.965473 16960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:19.965543 16960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:19.968582 16960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:19.968645 16960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:19.968741 16960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:03:19.972782 16960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:45547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:41975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:19.973776 16960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:19.974606 16960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:19.982944 16968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:19.983814 16965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:19.984524 16966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:19.985345 16960 server_base.cc:1034] running on GCE node
I20250109 21:03:19.985689 16960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:19.986199 16960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:19.987346 16960 hybrid_clock.cc:648] HybridClock initialized: now 1736456599987331 us; error 33 us; skew 500 ppm
Jan 09 21:03:19 dist-test-slave-g0zb krb5kdc[16853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456599, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:19.989938 16960 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:03:19.990770 16960 webserver.cc:458] Webserver started at http://127.12.126.65:38449/ using document root <none> and password file <none>
I20250109 21:03:19.991309 16960 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:19.991367 16960 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:19.991555 16960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:19.993221 16960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "a9253e4659b24fd1a421ca692bb1b77e"
format_stamp: "Formatted at 2025-01-09 21:03:19 on dist-test-slave-g0zb"
server_key: "3791ec8102165ff6812c173fdd069478"
server_key_iv: "1483a2b4833b33b7f1b2015fac52406a"
server_key_version: "encryptionkey@0"
I20250109 21:03:19.993726 16960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "a9253e4659b24fd1a421ca692bb1b77e"
format_stamp: "Formatted at 2025-01-09 21:03:19 on dist-test-slave-g0zb"
server_key: "3791ec8102165ff6812c173fdd069478"
server_key_iv: "1483a2b4833b33b7f1b2015fac52406a"
server_key_version: "encryptionkey@0"
I20250109 21:03:19.997053 16960 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250109 21:03:19.999171 16975 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:20.000208 16960 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:20.000305 16960 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "a9253e4659b24fd1a421ca692bb1b77e"
format_stamp: "Formatted at 2025-01-09 21:03:19 on dist-test-slave-g0zb"
server_key: "3791ec8102165ff6812c173fdd069478"
server_key_iv: "1483a2b4833b33b7f1b2015fac52406a"
server_key_version: "encryptionkey@0"
I20250109 21:03:20.000386 16960 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:20.023869 16960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:20.026629 16960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:20.026793 16960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:20.027361 16960 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:20.028088 16960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:20.028142 16960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:20.028199 16960 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:20.028224 16960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:20.036500 16960 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:37977
I20250109 21:03:20.036612 17088 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:37977 every 8 connection(s)
I20250109 21:03:20.037365 16960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:03:20.044334 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 16960
I20250109 21:03:20.044471 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:03:20.044775 12793 external_mini_cluster.cc:1453] Setting key 1dbbc6ab283c75dcab063d15f72cbe52
Jan 09 21:03:20 dist-test-slave-g0zb krb5kdc[16853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456599, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:20.049000 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:20.038926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:36929 (local address 127.12.126.126:41975)
0109 21:03:20.039055 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:20.039059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:20.039679 (+ 620us) server_negotiation.cc:407] Connection header received
0109 21:03:20.040793 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:20.040796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:20.040842 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:20.040917 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:20.042017 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:20.042523 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:20.043161 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:20.043310 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:20.045514 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:20.045530 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:20.045533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:20.045557 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:20.047051 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:20.047613 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:20.047619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:20.047623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:20.047688 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:20.047932 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:20.047935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:20.047939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:20.048122 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:20.048220 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:20.048632 (+ 412us) server_negotiation.cc:299] Negotiation successful
0109 21:03:20.048767 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250109 21:03:20.049378 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:20.039173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:41975 (local address 127.12.126.65:36929)
0109 21:03:20.039553 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:20.039585 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:20.040614 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:20.041043 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:20.041050 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:20.041392 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:20.041898 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:20.041910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:20.042615 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:20.042617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:20.043060 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:20.043068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:20.043214 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:20.043701 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:20.043720 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:20.045351 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:20.047148 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:20.047155 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:20.047168 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:20.047508 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:20.047784 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:20.047786 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:20.047788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:20.047863 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:20.048211 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:20.048215 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:20.048410 (+ 195us) client_negotiation.cc:769] Sending connection context
0109 21:03:20.048564 (+ 154us) client_negotiation.cc:240] Negotiation successful
0109 21:03:20.048757 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":93,"threads_started":1}
I20250109 21:03:20.050417 17089 heartbeater.cc:346] Connected to a master server at 127.12.126.126:41975
I20250109 21:03:20.050607 17089 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:20.051069 17089 heartbeater.cc:510] Master 127.12.126.126:41975 requested a full tablet report, sending...
I20250109 21:03:20.052589 16898 ts_manager.cc:194] Registered new tserver with Master: a9253e4659b24fd1a421ca692bb1b77e (127.12.126.65:37977)
WARNING: no policy specified for kudu/127.12.126.66@KRBTEST.COM; defaulting to no policy
I20250109 21:03:20.054037 16898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:36929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
I20250109 21:03:20.069845 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:41975
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:45547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:20.172441 17094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:20.172632 17094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:20.172683 17094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:20.175720 17094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:20.175769 17094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:20.175837 17094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:03:20.179761 17094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:45547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:41975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:20.180670 17094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:20.181407 17094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:20.189456 17102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:20.190300 17100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:20.191131 17099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:20.191918 17094 server_base.cc:1034] running on GCE node
I20250109 21:03:20.192240 17094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:20.192729 17094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:20.193866 17094 hybrid_clock.cc:648] HybridClock initialized: now 1736456600193846 us; error 32 us; skew 500 ppm
Jan 09 21:03:20 dist-test-slave-g0zb krb5kdc[16853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456600, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:20.196350 17094 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:03:20.197147 17094 webserver.cc:458] Webserver started at http://127.12.126.66:32827/ using document root <none> and password file <none>
I20250109 21:03:20.197644 17094 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:20.197693 17094 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:20.197831 17094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:20.199458 17094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "de8d0e0bfa264ce5bb9ba60e08f0571c"
format_stamp: "Formatted at 2025-01-09 21:03:20 on dist-test-slave-g0zb"
server_key: "f1a449bb1f7e4c713f6d4335965fc484"
server_key_iv: "06e12f120e989669bd2ddda93b0e75bf"
server_key_version: "encryptionkey@0"
I20250109 21:03:20.199863 17094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "de8d0e0bfa264ce5bb9ba60e08f0571c"
format_stamp: "Formatted at 2025-01-09 21:03:20 on dist-test-slave-g0zb"
server_key: "f1a449bb1f7e4c713f6d4335965fc484"
server_key_iv: "06e12f120e989669bd2ddda93b0e75bf"
server_key_version: "encryptionkey@0"
I20250109 21:03:20.202978 17094 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250109 21:03:20.204977 17109 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:20.205870 17094 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:20.205957 17094 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "de8d0e0bfa264ce5bb9ba60e08f0571c"
format_stamp: "Formatted at 2025-01-09 21:03:20 on dist-test-slave-g0zb"
server_key: "f1a449bb1f7e4c713f6d4335965fc484"
server_key_iv: "06e12f120e989669bd2ddda93b0e75bf"
server_key_version: "encryptionkey@0"
I20250109 21:03:20.206027 17094 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:20.219028 17094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:20.221392 17094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:20.221529 17094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:20.221946 17094 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:20.222659 17094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:20.222704 17094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:20.222750 17094 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:20.222771 17094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:20.230865 17094 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:33529
I20250109 21:03:20.230885 17222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:33529 every 8 connection(s)
I20250109 21:03:20.231700 17094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:03:20.235035 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 17094
I20250109 21:03:20.235157 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:03:20.235472 12793 external_mini_cluster.cc:1453] Setting key db8e63913554665b1547691fbc75eeae
Jan 09 21:03:20 dist-test-slave-g0zb krb5kdc[16853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456600, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:20.244277 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:20.233216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:34893 (local address 127.12.126.126:41975)
0109 21:03:20.233333 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:20.233337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:20.233964 (+ 627us) server_negotiation.cc:407] Connection header received
0109 21:03:20.235227 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:20.235231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:20.235294 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:20.235405 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:20.236639 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:20.237371 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:20.237948 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:20.238162 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:20.240550 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:20.240571 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:20.240576 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:20.240606 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:20.242348 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:20.242828 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:20.242833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:20.242837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:20.242891 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:20.243134 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:20.243139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:20.243141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:20.243346 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:20.243450 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:20.243919 (+ 469us) server_negotiation.cc:299] Negotiation successful
0109 21:03:20.244053 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
WARNING: no policy specified for kudu/127.12.126.67@KRBTEST.COM; defaulting to no policy
I20250109 21:03:20.244736 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:20.233444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:41975 (local address 127.12.126.66:34893)
0109 21:03:20.233829 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:20.233862 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:20.235019 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:20.235533 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:20.235545 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:20.235953 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:20.236504 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:20.236517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:20.237483 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:20.237485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:20.237852 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:20.237858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:20.238012 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:20.238586 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:20.238621 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:20.240383 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:20.242469 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:20.242473 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:20.242483 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:20.242714 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:20.242973 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:20.242975 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:20.242977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:20.243052 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:20.243453 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:20.243457 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:20.243662 (+ 205us) client_negotiation.cc:769] Sending connection context
0109 21:03:20.243828 (+ 166us) client_negotiation.cc:240] Negotiation successful
0109 21:03:20.244048 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":91,"threads_started":1}
I20250109 21:03:20.245824 17223 heartbeater.cc:346] Connected to a master server at 127.12.126.126:41975
I20250109 21:03:20.246028 17223 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:20.246557 17223 heartbeater.cc:510] Master 127.12.126.126:41975 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.67@KRBTEST.COM" created.
I20250109 21:03:20.247637 16898 ts_manager.cc:194] Registered new tserver with Master: de8d0e0bfa264ce5bb9ba60e08f0571c (127.12.126.66:33529)
I20250109 21:03:20.248383 16898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:34893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
I20250109 21:03:20.259994 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:41975
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:45547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:20.359678 17228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:20.359869 17228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:20.359936 17228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:20.363030 17228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:20.363080 17228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:20.363183 17228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:03:20.367179 17228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:45547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:41975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:20.368149 17228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:20.368919 17228 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:20.377190 17233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:20.377826 17236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:20.378695 17234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:20.379456 17228 server_base.cc:1034] running on GCE node
I20250109 21:03:20.379840 17228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:20.380371 17228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:20.381584 17228 hybrid_clock.cc:648] HybridClock initialized: now 1736456600381564 us; error 33 us; skew 500 ppm
Jan 09 21:03:20 dist-test-slave-g0zb krb5kdc[16853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456600, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:20.384107 17228 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:03:20.384948 17228 webserver.cc:458] Webserver started at http://127.12.126.67:37953/ using document root <none> and password file <none>
I20250109 21:03:20.385463 17228 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:20.385515 17228 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:20.385682 17228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:20.387380 17228 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "072b93e925764ab4bf4569ab24dbbda4"
format_stamp: "Formatted at 2025-01-09 21:03:20 on dist-test-slave-g0zb"
server_key: "688a181a8ac01e8134343e12ad2faf57"
server_key_iv: "85f65dd1a27622d2998ac14d80f8837d"
server_key_version: "encryptionkey@0"
I20250109 21:03:20.387823 17228 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "072b93e925764ab4bf4569ab24dbbda4"
format_stamp: "Formatted at 2025-01-09 21:03:20 on dist-test-slave-g0zb"
server_key: "688a181a8ac01e8134343e12ad2faf57"
server_key_iv: "85f65dd1a27622d2998ac14d80f8837d"
server_key_version: "encryptionkey@0"
I20250109 21:03:20.391010 17228 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:03:20.393002 17243 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:20.393903 17228 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:20.394008 17228 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "072b93e925764ab4bf4569ab24dbbda4"
format_stamp: "Formatted at 2025-01-09 21:03:20 on dist-test-slave-g0zb"
server_key: "688a181a8ac01e8134343e12ad2faf57"
server_key_iv: "85f65dd1a27622d2998ac14d80f8837d"
server_key_version: "encryptionkey@0"
I20250109 21:03:20.394101 17228 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:20.428222 17228 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:20.430668 17228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:20.430823 17228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:20.431260 17228 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:20.431954 17228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:20.431996 17228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:20.432034 17228 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:20.432054 17228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:20.439865 17228 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:35291
I20250109 21:03:20.439893 17356 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:35291 every 8 connection(s)
I20250109 21:03:20.440743 17228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:03:20.444794 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 17228
I20250109 21:03:20.444900 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:03:20.445122 12793 external_mini_cluster.cc:1453] Setting key 42a03230a0ea34ab1e1e14388705857d
Jan 09 21:03:20 dist-test-slave-g0zb krb5kdc[16853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456600, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:20.451786 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:20.442313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:41615 (local address 127.12.126.126:41975)
0109 21:03:20.442465 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:20.442471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:20.443112 (+ 641us) server_negotiation.cc:407] Connection header received
0109 21:03:20.444295 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:20.444298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:20.444347 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:20.444421 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:20.445707 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:20.446354 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:20.447070 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:20.447195 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:20.449155 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:20.449170 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:20.449172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:20.449197 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:20.450367 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:20.450749 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:20.450752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:20.450753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:20.450785 (+ 32us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:20.451005 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:20.451007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:20.451008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:20.451133 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:20.451202 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:20.451572 (+ 370us) server_negotiation.cc:299] Negotiation successful
0109 21:03:20.451672 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250109 21:03:20.452380 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:20.442577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:41975 (local address 127.12.126.67:41615)
0109 21:03:20.442986 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:20.443018 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:20.444081 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:20.444550 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:20.444557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:20.444933 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:20.445579 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:20.445593 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:20.446456 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:20.446459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:20.446980 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:20.446989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:20.447142 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:20.447652 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:20.447674 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:20.449011 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:20.450462 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:20.450466 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:20.450473 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:20.450682 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:20.450871 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:20.450873 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:20.450875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:20.450943 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:20.451201 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:20.451205 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:20.451393 (+ 188us) client_negotiation.cc:769] Sending connection context
0109 21:03:20.451543 (+ 150us) client_negotiation.cc:240] Negotiation successful
0109 21:03:20.451748 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":95,"threads_started":1}
I20250109 21:03:20.453301 17357 heartbeater.cc:346] Connected to a master server at 127.12.126.126:41975
I20250109 21:03:20.453486 17357 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:20.453892 17357 heartbeater.cc:510] Master 127.12.126.126:41975 requested a full tablet report, sending...
I20250109 21:03:20.454742 16898 ts_manager.cc:194] Registered new tserver with Master: 072b93e925764ab4bf4569ab24dbbda4 (127.12.126.67:35291)
I20250109 21:03:20.455265 16898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:41615
I20250109 21:03:20.458247 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
Jan 09 21:03:20 dist-test-slave-g0zb krb5kdc[16853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 09 21:03:20 dist-test-slave-g0zb krb5kdc[16853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.65@KRBTEST.COM
I20250109 21:03:20.480214 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:20.472502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38816 (local address 127.12.126.65:37977)
0109 21:03:20.472819 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:20.472825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:20.472847 (+ 22us) server_negotiation.cc:407] Connection header received
0109 21:03:20.472897 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:20.472901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:20.473061 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:20.473187 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:20.473924 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:20.474621 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:20.475227 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:20.475413 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:20.477251 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:20.477272 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:20.477281 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:20.477303 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:20.478905 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:20.479230 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:20.479234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:20.479235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:20.479279 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:20.479427 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:20.479429 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:20.479430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:20.479598 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:20.479826 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:20.479956 (+ 130us) server_negotiation.cc:299] Negotiation successful
0109 21:03:20.480065 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":139,"threads_started":1}
Jan 09 21:03:20 dist-test-slave-g0zb krb5kdc[16853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:20.489303 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:20.482633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55028 (local address 127.12.126.126:41975)
0109 21:03:20.482762 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:20.482769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:20.482818 (+ 49us) server_negotiation.cc:407] Connection header received
0109 21:03:20.482937 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:20.482940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:20.482984 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:20.483050 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:20.483676 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:20.484159 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:20.484724 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:20.484922 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:20.486746 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:20.486762 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:20.486764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:20.486785 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:20.487961 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:20.488300 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:20.488304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:20.488306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:20.488345 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:20.488554 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:20.488558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:20.488559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:20.488713 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:20.488839 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:20.489026 (+ 187us) server_negotiation.cc:299] Negotiation successful
0109 21:03:20.489131 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20250109 21:03:20.489639 16898 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55028
I20250109 21:03:20.490190 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 16960
I20250109 21:03:20.496490 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 17094
I20250109 21:03:20.502022 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 17228
I20250109 21:03:20.507275 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 16866
2025-01-09T21:03:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2070 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:03:20 dist-test-slave-g0zb krb5kdc[17369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:03:20 dist-test-slave-g0zb krb5kdc[17369](info): set up 2 sockets
Jan 09 21:03:20 dist-test-slave-g0zb krb5kdc[17369](info): commencing operation
krb5kdc: starting...
W20250109 21:03:21.511539 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:03:21 dist-test-slave-g0zb krb5kdc[17369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:03:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:03:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:03:21.619393 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:40361
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:40361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:21.720144 17382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:21.720378 17382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:21.720434 17382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:21.723831 17382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:21.723894 17382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:21.723917 17382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:21.723945 17382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:21.723968 17382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:03:21.728410 17382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:40361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:40361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:21.729454 17382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:21.730268 17382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:21.737864 17388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:21.738721 17387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:21.739502 17390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:21.740338 17382 server_base.cc:1034] running on GCE node
I20250109 21:03:21.740769 17382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:21.741590 17382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:21.742741 17382 hybrid_clock.cc:648] HybridClock initialized: now 1736456601742718 us; error 40 us; skew 500 ppm
Jan 09 21:03:21 dist-test-slave-g0zb krb5kdc[17369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456601, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:21.750334 17382 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:03:21.751111 17382 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:21.751173 17382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:21.751367 17382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:21.753130 17382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "8968e9633a3c4b6e9809c98f12be40f0"
format_stamp: "Formatted at 2025-01-09 21:03:21 on dist-test-slave-g0zb"
server_key: "b8722ae9db83697010fe138cc8a8d080"
server_key_iv: "c8be9d4666e751c3a760d509bc399831"
server_key_version: "encryptionkey@0"
I20250109 21:03:21.753631 17382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "8968e9633a3c4b6e9809c98f12be40f0"
format_stamp: "Formatted at 2025-01-09 21:03:21 on dist-test-slave-g0zb"
server_key: "b8722ae9db83697010fe138cc8a8d080"
server_key_iv: "c8be9d4666e751c3a760d509bc399831"
server_key_version: "encryptionkey@0"
I20250109 21:03:21.756896 17382 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250109 21:03:21.758872 17396 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:21.759794 17382 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:21.759896 17382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "8968e9633a3c4b6e9809c98f12be40f0"
format_stamp: "Formatted at 2025-01-09 21:03:21 on dist-test-slave-g0zb"
server_key: "b8722ae9db83697010fe138cc8a8d080"
server_key_iv: "c8be9d4666e751c3a760d509bc399831"
server_key_version: "encryptionkey@0"
I20250109 21:03:21.759981 17382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:21.772359 17382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:21.774789 17382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:21.774952 17382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:21.780601 17448 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:40361 every 8 connection(s)
I20250109 21:03:21.780612 17382 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:40361
I20250109 21:03:21.781141 17382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:03:21.783756 17449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:21.784185 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 17382
I20250109 21:03:21.784303 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:03:21.784560 12793 external_mini_cluster.cc:1453] Setting key 925800c3f1a9435a3ad439a6e282faaa
I20250109 21:03:21.789410 17449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0: Bootstrap starting.
I20250109 21:03:21.791479 17449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0: Neither blocks nor log segments found. Creating new log.
Jan 09 21:03:21 dist-test-slave-g0zb krb5kdc[17369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:21.792282 17449 log.cc:826] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:21.794039 17449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0: No bootstrap required, opened a new log
I20250109 21:03:21.796587 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:21.785688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.12.126.126:40361)
0109 21:03:21.786064 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:21.786076 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:21.786142 (+ 66us) server_negotiation.cc:407] Connection header received
0109 21:03:21.786886 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:21.786909 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:21.787244 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:21.787605 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:21.788307 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:21.789396 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:21.790017 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:21.790278 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:21.792467 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:21.792505 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:21.792521 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:21.792565 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:21.794306 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:21.794761 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:21.794765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:21.794769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:21.794874 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:21.795098 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:21.795100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:21.795101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:21.795396 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:21.795526 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:21.795802 (+ 276us) server_negotiation.cc:299] Negotiation successful
0109 21:03:21.796014 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":101,"threads_started":1}
I20250109 21:03:21.797011 17449 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8968e9633a3c4b6e9809c98f12be40f0" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40361 } }
I20250109 21:03:21.797178 17449 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:21.797240 17449 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8968e9633a3c4b6e9809c98f12be40f0, State: Initialized, Role: FOLLOWER
I20250109 21:03:21.797672 17449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8968e9633a3c4b6e9809c98f12be40f0" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40361 } }
I20250109 21:03:21.797814 17449 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:03:21.797873 17449 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:03:21.797976 17449 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:21.798727 17449 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8968e9633a3c4b6e9809c98f12be40f0" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40361 } }
I20250109 21:03:21.799014 17449 leader_election.cc:304] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8968e9633a3c4b6e9809c98f12be40f0; no voters:
I20250109 21:03:21.799221 17449 leader_election.cc:290] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:03:21.799310 17454 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:21.799523 17454 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0 [term 1 LEADER]: Becoming Leader. State: Replica: 8968e9633a3c4b6e9809c98f12be40f0, State: Running, Role: LEADER
I20250109 21:03:21.799831 17454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8968e9633a3c4b6e9809c98f12be40f0" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40361 } }
I20250109 21:03:21.800164 17449 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:21.801270 17456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8968e9633a3c4b6e9809c98f12be40f0. Latest consensus state: current_term: 1 leader_uuid: "8968e9633a3c4b6e9809c98f12be40f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8968e9633a3c4b6e9809c98f12be40f0" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40361 } } }
I20250109 21:03:21.801380 17456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:21.801738 17463 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:03:21.801728 17455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8968e9633a3c4b6e9809c98f12be40f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8968e9633a3c4b6e9809c98f12be40f0" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40361 } } }
I20250109 21:03:21.801906 17455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:21.804653 17463 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:03:21.809578 17463 catalog_manager.cc:1348] Generated new cluster ID: 61deff9bfd92460a9c032989f089379d
I20250109 21:03:21.809640 17463 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:03:21.817173 17463 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:03:21.818121 17463 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:03:21.827733 17463 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0: Generated new TSK 0
I20250109 21:03:21.828438 17463 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
I20250109 21:03:21.861655 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:40361
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:21.979997 17475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:21.980216 17475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:21.980293 17475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:21.983471 17475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:21.983529 17475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:21.983642 17475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:03:21.987707 17475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:40361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:21.988792 17475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:21.989621 17475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:21.998152 17480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:21.998905 17481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:21.999718 17483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:22.000535 17475 server_base.cc:1034] running on GCE node
I20250109 21:03:22.000869 17475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:22.001405 17475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:22.002599 17475 hybrid_clock.cc:648] HybridClock initialized: now 1736456602002583 us; error 37 us; skew 500 ppm
Jan 09 21:03:22 dist-test-slave-g0zb krb5kdc[17369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456602, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:22.005189 17475 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:03:22.005868 17475 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:22.005924 17475 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:22.006065 17475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:22.007809 17475 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "556f3282fa904792b4f25319a95bccc8"
format_stamp: "Formatted at 2025-01-09 21:03:22 on dist-test-slave-g0zb"
server_key: "dd36c990648f0af5712c5d8d48794128"
server_key_iv: "001bb1c5b49af70f243f82a3abbb5651"
server_key_version: "encryptionkey@0"
I20250109 21:03:22.008224 17475 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "556f3282fa904792b4f25319a95bccc8"
format_stamp: "Formatted at 2025-01-09 21:03:22 on dist-test-slave-g0zb"
server_key: "dd36c990648f0af5712c5d8d48794128"
server_key_iv: "001bb1c5b49af70f243f82a3abbb5651"
server_key_version: "encryptionkey@0"
I20250109 21:03:22.011428 17475 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:03:22.013509 17489 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:22.014420 17475 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:22.014555 17475 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "556f3282fa904792b4f25319a95bccc8"
format_stamp: "Formatted at 2025-01-09 21:03:22 on dist-test-slave-g0zb"
server_key: "dd36c990648f0af5712c5d8d48794128"
server_key_iv: "001bb1c5b49af70f243f82a3abbb5651"
server_key_version: "encryptionkey@0"
I20250109 21:03:22.014640 17475 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:22.037349 17475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:22.039949 17475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:22.040105 17475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:22.040576 17475 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:22.041314 17475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:22.041357 17475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:22.041400 17475 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:22.041429 17475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:22.049247 17475 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:34843
I20250109 21:03:22.049291 17602 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:34843 every 8 connection(s)
I20250109 21:03:22.049736 17475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
Jan 09 21:03:22 dist-test-slave-g0zb krb5kdc[17369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456602, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:22.057978 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 17475
I20250109 21:03:22.058102 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:03:22.058374 12793 external_mini_cluster.cc:1453] Setting key f71ce3ba4ea520df5b0677a762536b02
I20250109 21:03:22.061470 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.051220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:47391 (local address 127.12.126.126:40361)
0109 21:03:22.051345 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:22.051349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:22.051984 (+ 635us) server_negotiation.cc:407] Connection header received
0109 21:03:22.053200 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:22.053203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:22.053261 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:22.053337 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:22.054530 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.055063 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:22.055691 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.055862 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:22.057885 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:22.057906 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:22.057909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:22.057940 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:22.059479 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:22.059976 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:22.059980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:22.059982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:22.060045 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:22.060270 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:22.060275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:22.060286 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:22.060532 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:22.060636 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:22.061111 (+ 475us) server_negotiation.cc:299] Negotiation successful
0109 21:03:22.061242 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250109 21:03:22.062168 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.051460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:40361 (local address 127.12.126.65:47391)
0109 21:03:22.051862 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:22.051895 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:22.052973 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:22.053509 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:22.053516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:22.053847 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:22.054359 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:22.054371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.055182 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:22.055185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:22.055573 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:22.055581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.055786 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:22.056331 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:22.056353 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:22.057721 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:22.059589 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:22.059594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:22.059603 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:22.059887 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:22.060124 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:22.060125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:22.060127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:22.060195 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:22.060667 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:22.060672 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:22.060881 (+ 209us) client_negotiation.cc:769] Sending connection context
0109 21:03:22.061054 (+ 173us) client_negotiation.cc:240] Negotiation successful
0109 21:03:22.061302 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
I20250109 21:03:22.063234 17603 heartbeater.cc:346] Connected to a master server at 127.12.126.126:40361
I20250109 21:03:22.063436 17603 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:22.063839 17603 heartbeater.cc:510] Master 127.12.126.126:40361 requested a full tablet report, sending...
I20250109 21:03:22.065150 17413 ts_manager.cc:194] Registered new tserver with Master: 556f3282fa904792b4f25319a95bccc8 (127.12.126.65:34843)
I20250109 21:03:22.066350 17413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:47391
WARNING: no policy specified for kudu/127.12.126.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
I20250109 21:03:22.085671 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:40361
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:22.188390 17608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:22.188594 17608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:22.188647 17608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:22.191568 17608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:22.191617 17608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:22.191689 17608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:03:22.195938 17608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:40361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:22.196884 17608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:22.197664 17608 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:22.205334 17613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:22.206225 17616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:22.206941 17614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:22.207819 17608 server_base.cc:1034] running on GCE node
I20250109 21:03:22.208168 17608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:22.208647 17608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:22.209784 17608 hybrid_clock.cc:648] HybridClock initialized: now 1736456602209767 us; error 29 us; skew 500 ppm
Jan 09 21:03:22 dist-test-slave-g0zb krb5kdc[17369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456602, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:22.212236 17608 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:03:22.212888 17608 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:22.212939 17608 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:22.213101 17608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:22.214795 17608 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "cee2e84160c4418cbb2ad8deac315f3b"
format_stamp: "Formatted at 2025-01-09 21:03:22 on dist-test-slave-g0zb"
server_key: "1c2664645c669a2372db64c2a0a16a16"
server_key_iv: "b8825bb16b9d470b537fc64982d4c53d"
server_key_version: "encryptionkey@0"
I20250109 21:03:22.215289 17608 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "cee2e84160c4418cbb2ad8deac315f3b"
format_stamp: "Formatted at 2025-01-09 21:03:22 on dist-test-slave-g0zb"
server_key: "1c2664645c669a2372db64c2a0a16a16"
server_key_iv: "b8825bb16b9d470b537fc64982d4c53d"
server_key_version: "encryptionkey@0"
I20250109 21:03:22.218318 17608 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:03:22.220448 17622 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:22.221330 17608 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:03:22.221454 17608 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "cee2e84160c4418cbb2ad8deac315f3b"
format_stamp: "Formatted at 2025-01-09 21:03:22 on dist-test-slave-g0zb"
server_key: "1c2664645c669a2372db64c2a0a16a16"
server_key_iv: "b8825bb16b9d470b537fc64982d4c53d"
server_key_version: "encryptionkey@0"
I20250109 21:03:22.221531 17608 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:22.249552 17608 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:22.252242 17608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:22.252425 17608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:22.252931 17608 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:22.253664 17608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:22.253707 17608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:22.253760 17608 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:22.253790 17608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:22.262194 17608 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:44843
I20250109 21:03:22.262212 17735 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:44843 every 8 connection(s)
I20250109 21:03:22.262712 17608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
Jan 09 21:03:22 dist-test-slave-g0zb krb5kdc[17369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456602, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:22.270956 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 17608
I20250109 21:03:22.271064 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:03:22.271301 12793 external_mini_cluster.cc:1453] Setting key 360c4e4e764cb00958f14ee88a8b403c
I20250109 21:03:22.275003 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.264204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:56917 (local address 127.12.126.126:40361)
0109 21:03:22.264340 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:22.264344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:22.265071 (+ 727us) server_negotiation.cc:407] Connection header received
0109 21:03:22.266282 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:22.266285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:22.266348 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:22.266424 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:22.267706 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.268220 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:22.268841 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.269052 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:22.271248 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:22.271265 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:22.271267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:22.271292 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:22.272932 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:22.273514 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:22.273517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:22.273519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:22.273572 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:22.273869 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:22.273872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:22.273873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:22.274003 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:22.274112 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:22.274666 (+ 554us) server_negotiation.cc:299] Negotiation successful
0109 21:03:22.274836 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20250109 21:03:22.275630 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.264505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:40361 (local address 127.12.126.66:56917)
0109 21:03:22.264932 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:22.264965 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:22.266082 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:22.266594 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:22.266601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:22.267002 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:22.267558 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:22.267570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.268337 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:22.268339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:22.268739 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:22.268746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.268960 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:22.269662 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:22.269684 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:22.271099 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:22.273060 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:22.273069 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:22.273081 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:22.273417 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:22.273645 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:22.273651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:22.273655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:22.273763 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:22.274112 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:22.274118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:22.274386 (+ 268us) client_negotiation.cc:769] Sending connection context
0109 21:03:22.274593 (+ 207us) client_negotiation.cc:240] Negotiation successful
0109 21:03:22.274790 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":102,"threads_started":1}
I20250109 21:03:22.276690 17736 heartbeater.cc:346] Connected to a master server at 127.12.126.126:40361
I20250109 21:03:22.276932 17736 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:22.277384 17736 heartbeater.cc:510] Master 127.12.126.126:40361 requested a full tablet report, sending...
I20250109 21:03:22.278348 17413 ts_manager.cc:194] Registered new tserver with Master: cee2e84160c4418cbb2ad8deac315f3b (127.12.126.66:44843)
I20250109 21:03:22.278913 17413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:56917
WARNING: no policy specified for kudu/127.12.126.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
I20250109 21:03:22.295790 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:40361
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:22.393730 17741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:22.393985 17741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:22.394080 17741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:22.397133 17741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:22.397199 17741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:22.397292 17741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:03:22.401101 17741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:40361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:22.401947 17741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:22.402757 17741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:22.410511 17749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:22.411360 17746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:22.412123 17747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:22.412937 17741 server_base.cc:1034] running on GCE node
I20250109 21:03:22.413218 17741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:22.413662 17741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:22.414809 17741 hybrid_clock.cc:648] HybridClock initialized: now 1736456602414780 us; error 27 us; skew 500 ppm
Jan 09 21:03:22 dist-test-slave-g0zb krb5kdc[17369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456602, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:22.417311 17741 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:03:22.417945 17741 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:22.418004 17741 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:22.418159 17741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:22.419864 17741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "a89c6dd645ff4d2da0d8dc7c51abd383"
format_stamp: "Formatted at 2025-01-09 21:03:22 on dist-test-slave-g0zb"
server_key: "0947e9b2cac98998e4223c07fc4b95a6"
server_key_iv: "1d169e118d611ce4a84695e104a3c267"
server_key_version: "encryptionkey@0"
I20250109 21:03:22.420327 17741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "a89c6dd645ff4d2da0d8dc7c51abd383"
format_stamp: "Formatted at 2025-01-09 21:03:22 on dist-test-slave-g0zb"
server_key: "0947e9b2cac98998e4223c07fc4b95a6"
server_key_iv: "1d169e118d611ce4a84695e104a3c267"
server_key_version: "encryptionkey@0"
I20250109 21:03:22.423427 17741 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:03:22.425411 17755 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:22.426311 17741 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:22.426406 17741 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "a89c6dd645ff4d2da0d8dc7c51abd383"
format_stamp: "Formatted at 2025-01-09 21:03:22 on dist-test-slave-g0zb"
server_key: "0947e9b2cac98998e4223c07fc4b95a6"
server_key_iv: "1d169e118d611ce4a84695e104a3c267"
server_key_version: "encryptionkey@0"
I20250109 21:03:22.426477 17741 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:22.449673 17741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:22.452783 17741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:22.452940 17741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:22.453418 17741 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:22.454139 17741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:22.454185 17741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:22.454253 17741 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:22.454278 17741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:22.462489 17741 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:44639
I20250109 21:03:22.462515 17868 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:44639 every 8 connection(s)
I20250109 21:03:22.463021 17741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
Jan 09 21:03:22 dist-test-slave-g0zb krb5kdc[17369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456602, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:22.470825 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 17741
I20250109 21:03:22.470925 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:03:22.471192 12793 external_mini_cluster.cc:1453] Setting key 236dc398e0e3a3b2ce08162dd661bf8c
I20250109 21:03:22.474141 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.464559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:34637 (local address 127.12.126.126:40361)
0109 21:03:22.464682 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:22.464686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:22.465310 (+ 624us) server_negotiation.cc:407] Connection header received
0109 21:03:22.466488 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:22.466490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:22.466536 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:22.466608 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:22.467776 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.468262 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:22.468857 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.468974 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:22.470991 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:22.471013 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:22.471015 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:22.471048 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:22.472551 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:22.473041 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:22.473044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:22.473046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:22.473095 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:22.473353 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:22.473355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:22.473357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:22.473486 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:22.473567 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:22.473945 (+ 378us) server_negotiation.cc:299] Negotiation successful
0109 21:03:22.474032 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250109 21:03:22.474718 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.464797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:40361 (local address 127.12.126.67:34637)
0109 21:03:22.465184 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:22.465217 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:22.466321 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:22.466778 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:22.466786 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:22.467145 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:22.467664 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:22.467677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.468374 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:22.468377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:22.468766 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:22.468774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.468933 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:22.469425 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:22.469446 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:22.470829 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:22.472650 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:22.472655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:22.472664 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:22.472954 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:22.473200 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:22.473202 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:22.473204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:22.473282 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:22.473578 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:22.473583 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:22.473773 (+ 190us) client_negotiation.cc:769] Sending connection context
0109 21:03:22.473920 (+ 147us) client_negotiation.cc:240] Negotiation successful
0109 21:03:22.474102 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":100,"threads_started":1}
I20250109 21:03:22.475662 17869 heartbeater.cc:346] Connected to a master server at 127.12.126.126:40361
I20250109 21:03:22.475883 17869 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:22.476290 17869 heartbeater.cc:510] Master 127.12.126.126:40361 requested a full tablet report, sending...
I20250109 21:03:22.477130 17413 ts_manager.cc:194] Registered new tserver with Master: a89c6dd645ff4d2da0d8dc7c51abd383 (127.12.126.67:44639)
I20250109 21:03:22.477607 17413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:34637
I20250109 21:03:22.484650 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 21:03:22.492106 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.486415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.12.126.126:40361)
0109 21:03:22.486526 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:22.486529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:22.486652 (+ 123us) server_negotiation.cc:407] Connection header received
0109 21:03:22.486746 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:22.486748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:22.486783 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:22.486869 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:22.487579 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.488025 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:22.488570 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.488672 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:22.489658 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:22.489671 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:22.489673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:22.489693 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:22.491001 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:22.491356 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:22.491358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:22.491359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:22.491391 (+ 32us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:22.491577 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:22.491578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:22.491580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:22.491705 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:22.491797 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:22.491932 (+ 135us) server_negotiation.cc:299] Negotiation successful
0109 21:03:22.492009 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250109 21:03:22.494589 17413 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:03:22.497051 17413 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:03:22.509027 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.505007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:44639 (local address 127.0.0.1:41202)
0109 21:03:22.505466 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:22.505482 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:22.505569 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:22.506514 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:22.506517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:22.506535 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:22.506731 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:22.506738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.507695 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:22.507697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:22.508530 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:22.508539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.508638 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:22.508691 (+ 53us) client_negotiation.cc:769] Sending connection context
0109 21:03:22.508787 (+ 96us) client_negotiation.cc:240] Negotiation successful
0109 21:03:22.508883 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":87,"threads_started":1}
I20250109 21:03:22.509342 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.504753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:44843 (local address 127.0.0.1:49650)
0109 21:03:22.505322 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:22.505361 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:22.505505 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:22.506676 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:22.506679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:22.506693 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:22.506919 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:22.506930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.508193 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:22.508198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:22.509012 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:22.509022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.509121 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:22.509142 (+ 21us) client_negotiation.cc:769] Sending connection context
0109 21:03:22.509197 (+ 55us) client_negotiation.cc:240] Negotiation successful
0109 21:03:22.509254 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"spinlock_wait_cycles":4480,"thread_start_us":93,"threads_started":1}
I20250109 21:03:22.510054 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.504748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46554 (local address 127.12.126.65:34843)
0109 21:03:22.505125 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:22.505132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:22.505638 (+ 506us) server_negotiation.cc:407] Connection header received
0109 21:03:22.505717 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:22.505721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:22.505896 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:22.506043 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:22.506623 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.507745 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:22.508968 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.509659 (+ 691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:22.509773 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:22.509843 (+ 70us) server_negotiation.cc:299] Negotiation successful
0109 21:03:22.509926 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":128,"threads_started":1}
I20250109 21:03:22.510109 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.505665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.12.126.67:44639)
0109 21:03:22.506013 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:22.506018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:22.506037 (+ 19us) server_negotiation.cc:407] Connection header received
0109 21:03:22.506111 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:22.506115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:22.506262 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:22.506392 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:22.506882 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.507584 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:22.509318 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.509776 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:22.509862 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:22.509915 (+ 53us) server_negotiation.cc:299] Negotiation successful
0109 21:03:22.509983 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":117,"threads_started":1}
I20250109 21:03:22.510309 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.505332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49650 (local address 127.12.126.66:44843)
0109 21:03:22.506151 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:22.506166 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:22.506185 (+ 19us) server_negotiation.cc:407] Connection header received
0109 21:03:22.506266 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:22.506271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:22.506518 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:22.506687 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:22.507020 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.508066 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:22.509617 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.509954 (+ 337us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:22.510028 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:22.510077 (+ 49us) server_negotiation.cc:299] Negotiation successful
0109 21:03:22.510152 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":117,"threads_started":1}
I20250109 21:03:22.510363 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.504616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:34843 (local address 127.0.0.1:46554)
0109 21:03:22.505344 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:22.505361 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:22.505493 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:22.506211 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:22.506216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:22.506234 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:22.506506 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:22.506513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.507851 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:22.507854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:22.508627 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:22.508635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.508717 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:22.508728 (+ 11us) client_negotiation.cc:769] Sending connection context
0109 21:03:22.508787 (+ 59us) client_negotiation.cc:240] Negotiation successful
0109 21:03:22.508883 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"spinlock_wait_cycles":1408,"thread_start_us":68,"threads_started":1}
I20250109 21:03:22.512570 17803 tablet_service.cc:1467] Processing CreateTablet for tablet e32cd134bd6f47799f1c286b57d6cf6f (DEFAULT_TABLE table=test-table [id=842e5d9ef8f746fe9ca1ccbf9fe91cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:22.512570 17670 tablet_service.cc:1467] Processing CreateTablet for tablet e32cd134bd6f47799f1c286b57d6cf6f (DEFAULT_TABLE table=test-table [id=842e5d9ef8f746fe9ca1ccbf9fe91cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:22.513512 17670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e32cd134bd6f47799f1c286b57d6cf6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:22.513509 17803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e32cd134bd6f47799f1c286b57d6cf6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:22.516250 17537 tablet_service.cc:1467] Processing CreateTablet for tablet e32cd134bd6f47799f1c286b57d6cf6f (DEFAULT_TABLE table=test-table [id=842e5d9ef8f746fe9ca1ccbf9fe91cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:22.516978 17537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e32cd134bd6f47799f1c286b57d6cf6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:22.519871 17886 tablet_bootstrap.cc:492] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b: Bootstrap starting.
I20250109 21:03:22.520253 17887 tablet_bootstrap.cc:492] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383: Bootstrap starting.
I20250109 21:03:22.522418 17887 tablet_bootstrap.cc:654] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:22.522840 17888 tablet_bootstrap.cc:492] T e32cd134bd6f47799f1c286b57d6cf6f P 556f3282fa904792b4f25319a95bccc8: Bootstrap starting.
I20250109 21:03:22.524120 17887 log.cc:826] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:22.524436 17886 tablet_bootstrap.cc:654] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:22.524829 17888 tablet_bootstrap.cc:654] T e32cd134bd6f47799f1c286b57d6cf6f P 556f3282fa904792b4f25319a95bccc8: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:22.525676 17888 log.cc:826] T e32cd134bd6f47799f1c286b57d6cf6f P 556f3282fa904792b4f25319a95bccc8: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:22.525676 17886 log.cc:826] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:22.525751 17887 tablet_bootstrap.cc:492] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383: No bootstrap required, opened a new log
I20250109 21:03:22.525945 17887 ts_tablet_manager.cc:1397] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250109 21:03:22.527307 17886 tablet_bootstrap.cc:492] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b: No bootstrap required, opened a new log
I20250109 21:03:22.527388 17888 tablet_bootstrap.cc:492] T e32cd134bd6f47799f1c286b57d6cf6f P 556f3282fa904792b4f25319a95bccc8: No bootstrap required, opened a new log
I20250109 21:03:22.527479 17886 ts_tablet_manager.cc:1397] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20250109 21:03:22.527559 17888 ts_tablet_manager.cc:1397] T e32cd134bd6f47799f1c286b57d6cf6f P 556f3282fa904792b4f25319a95bccc8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250109 21:03:22.529130 17887 raft_consensus.cc:357] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556f3282fa904792b4f25319a95bccc8" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34843 } } peers { permanent_uuid: "cee2e84160c4418cbb2ad8deac315f3b" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44843 } } peers { permanent_uuid: "a89c6dd645ff4d2da0d8dc7c51abd383" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44639 } }
I20250109 21:03:22.529343 17887 raft_consensus.cc:383] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:22.529454 17887 raft_consensus.cc:738] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a89c6dd645ff4d2da0d8dc7c51abd383, State: Initialized, Role: FOLLOWER
I20250109 21:03:22.529888 17887 consensus_queue.cc:260] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556f3282fa904792b4f25319a95bccc8" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34843 } } peers { permanent_uuid: "cee2e84160c4418cbb2ad8deac315f3b" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44843 } } peers { permanent_uuid: "a89c6dd645ff4d2da0d8dc7c51abd383" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44639 } }
I20250109 21:03:22.530076 17886 raft_consensus.cc:357] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556f3282fa904792b4f25319a95bccc8" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34843 } } peers { permanent_uuid: "cee2e84160c4418cbb2ad8deac315f3b" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44843 } } peers { permanent_uuid: "a89c6dd645ff4d2da0d8dc7c51abd383" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44639 } }
I20250109 21:03:22.530284 17886 raft_consensus.cc:383] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:22.530377 17886 raft_consensus.cc:738] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee2e84160c4418cbb2ad8deac315f3b, State: Initialized, Role: FOLLOWER
I20250109 21:03:22.530648 17869 heartbeater.cc:502] Master 127.12.126.126:40361 was elected leader, sending a full tablet report...
I20250109 21:03:22.530814 17887 ts_tablet_manager.cc:1428] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:03:22.530833 17886 consensus_queue.cc:260] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556f3282fa904792b4f25319a95bccc8" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34843 } } peers { permanent_uuid: "cee2e84160c4418cbb2ad8deac315f3b" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44843 } } peers { permanent_uuid: "a89c6dd645ff4d2da0d8dc7c51abd383" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44639 } }
I20250109 21:03:22.531445 17736 heartbeater.cc:502] Master 127.12.126.126:40361 was elected leader, sending a full tablet report...
I20250109 21:03:22.531548 17886 ts_tablet_manager.cc:1428] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250109 21:03:22.531803 17888 raft_consensus.cc:357] T e32cd134bd6f47799f1c286b57d6cf6f P 556f3282fa904792b4f25319a95bccc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556f3282fa904792b4f25319a95bccc8" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34843 } } peers { permanent_uuid: "cee2e84160c4418cbb2ad8deac315f3b" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44843 } } peers { permanent_uuid: "a89c6dd645ff4d2da0d8dc7c51abd383" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44639 } }
I20250109 21:03:22.531939 17888 raft_consensus.cc:383] T e32cd134bd6f47799f1c286b57d6cf6f P 556f3282fa904792b4f25319a95bccc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:22.531975 17888 raft_consensus.cc:738] T e32cd134bd6f47799f1c286b57d6cf6f P 556f3282fa904792b4f25319a95bccc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556f3282fa904792b4f25319a95bccc8, State: Initialized, Role: FOLLOWER
I20250109 21:03:22.532305 17888 consensus_queue.cc:260] T e32cd134bd6f47799f1c286b57d6cf6f P 556f3282fa904792b4f25319a95bccc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556f3282fa904792b4f25319a95bccc8" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34843 } } peers { permanent_uuid: "cee2e84160c4418cbb2ad8deac315f3b" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44843 } } peers { permanent_uuid: "a89c6dd645ff4d2da0d8dc7c51abd383" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44639 } }
I20250109 21:03:22.532881 17603 heartbeater.cc:502] Master 127.12.126.126:40361 was elected leader, sending a full tablet report...
I20250109 21:03:22.533074 17888 ts_tablet_manager.cc:1428] T e32cd134bd6f47799f1c286b57d6cf6f P 556f3282fa904792b4f25319a95bccc8: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
W20250109 21:03:22.550690 17604 tablet.cc:2367] T e32cd134bd6f47799f1c286b57d6cf6f P 556f3282fa904792b4f25319a95bccc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250109 21:03:22.713864 17870 tablet.cc:2367] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:03:22.728303 17892 raft_consensus.cc:491] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:03:22.728504 17892 raft_consensus.cc:513] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556f3282fa904792b4f25319a95bccc8" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34843 } } peers { permanent_uuid: "cee2e84160c4418cbb2ad8deac315f3b" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44843 } } peers { permanent_uuid: "a89c6dd645ff4d2da0d8dc7c51abd383" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44639 } }
I20250109 21:03:22.729560 17892 leader_election.cc:290] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 556f3282fa904792b4f25319a95bccc8 (127.12.126.65:34843), cee2e84160c4418cbb2ad8deac315f3b (127.12.126.66:44843)
I20250109 21:03:22.732390 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.729782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:34843 (local address 127.12.126.67:42555)
0109 21:03:22.730011 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:22.730026 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:22.730127 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:22.730423 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:22.730426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:22.730443 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:22.730715 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:22.730724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.731476 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:22.731482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:22.732049 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:22.732057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.732148 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:22.732165 (+ 17us) client_negotiation.cc:769] Sending connection context
0109 21:03:22.732213 (+ 48us) client_negotiation.cc:240] Negotiation successful
0109 21:03:22.732267 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20250109 21:03:22.732555 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.729782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:44843 (local address 127.12.126.67:35237)
0109 21:03:22.730189 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:22.730204 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:22.730323 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:22.730580 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:22.730584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:22.730602 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:22.730874 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:22.730885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.731573 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:22.731575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:22.732117 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:22.732128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.732235 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:22.732253 (+ 18us) client_negotiation.cc:769] Sending connection context
0109 21:03:22.732301 (+ 48us) client_negotiation.cc:240] Negotiation successful
0109 21:03:22.732354 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":54,"thread_start_us":111,"threads_started":1}
I20250109 21:03:22.732906 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.729869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:42555 (local address 127.12.126.65:34843)
0109 21:03:22.729997 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:22.730002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:22.730100 (+ 98us) server_negotiation.cc:407] Connection header received
0109 21:03:22.730260 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:22.730267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:22.730334 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:22.730411 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:22.730853 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.731377 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:22.732175 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.732626 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:22.732673 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:22.732724 (+ 51us) server_negotiation.cc:299] Negotiation successful
0109 21:03:22.732786 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250109 21:03:22.732946 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.729874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:35237 (local address 127.12.126.66:44843)
0109 21:03:22.729997 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:22.730002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:22.730286 (+ 284us) server_negotiation.cc:407] Connection header received
0109 21:03:22.730432 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:22.730435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:22.730482 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:22.730583 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:22.730991 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.731443 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:22.732256 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.732699 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:22.732738 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:22.732793 (+ 55us) server_negotiation.cc:299] Negotiation successful
0109 21:03:22.732860 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250109 21:03:22.733485 17557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32cd134bd6f47799f1c286b57d6cf6f" candidate_uuid: "a89c6dd645ff4d2da0d8dc7c51abd383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "556f3282fa904792b4f25319a95bccc8" is_pre_election: true
I20250109 21:03:22.733486 17690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32cd134bd6f47799f1c286b57d6cf6f" candidate_uuid: "a89c6dd645ff4d2da0d8dc7c51abd383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee2e84160c4418cbb2ad8deac315f3b" is_pre_election: true
I20250109 21:03:22.733757 17690 raft_consensus.cc:2463] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a89c6dd645ff4d2da0d8dc7c51abd383 in term 0.
I20250109 21:03:22.733757 17557 raft_consensus.cc:2463] T e32cd134bd6f47799f1c286b57d6cf6f P 556f3282fa904792b4f25319a95bccc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a89c6dd645ff4d2da0d8dc7c51abd383 in term 0.
I20250109 21:03:22.734266 17759 leader_election.cc:304] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 556f3282fa904792b4f25319a95bccc8, a89c6dd645ff4d2da0d8dc7c51abd383, cee2e84160c4418cbb2ad8deac315f3b; no voters:
I20250109 21:03:22.734551 17892 raft_consensus.cc:2798] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:03:22.734625 17892 raft_consensus.cc:491] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:03:22.734685 17892 raft_consensus.cc:3054] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:22.735675 17892 raft_consensus.cc:513] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556f3282fa904792b4f25319a95bccc8" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34843 } } peers { permanent_uuid: "cee2e84160c4418cbb2ad8deac315f3b" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44843 } } peers { permanent_uuid: "a89c6dd645ff4d2da0d8dc7c51abd383" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44639 } }
I20250109 21:03:22.736052 17892 leader_election.cc:290] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [CANDIDATE]: Term 1 election: Requested vote from peers 556f3282fa904792b4f25319a95bccc8 (127.12.126.65:34843), cee2e84160c4418cbb2ad8deac315f3b (127.12.126.66:44843)
I20250109 21:03:22.736461 17557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32cd134bd6f47799f1c286b57d6cf6f" candidate_uuid: "a89c6dd645ff4d2da0d8dc7c51abd383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "556f3282fa904792b4f25319a95bccc8"
I20250109 21:03:22.736487 17690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32cd134bd6f47799f1c286b57d6cf6f" candidate_uuid: "a89c6dd645ff4d2da0d8dc7c51abd383" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee2e84160c4418cbb2ad8deac315f3b"
I20250109 21:03:22.736585 17557 raft_consensus.cc:3054] T e32cd134bd6f47799f1c286b57d6cf6f P 556f3282fa904792b4f25319a95bccc8 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:22.736610 17690 raft_consensus.cc:3054] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:22.737476 17690 raft_consensus.cc:2463] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a89c6dd645ff4d2da0d8dc7c51abd383 in term 1.
I20250109 21:03:22.737494 17557 raft_consensus.cc:2463] T e32cd134bd6f47799f1c286b57d6cf6f P 556f3282fa904792b4f25319a95bccc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a89c6dd645ff4d2da0d8dc7c51abd383 in term 1.
I20250109 21:03:22.737753 17756 leader_election.cc:304] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a89c6dd645ff4d2da0d8dc7c51abd383, cee2e84160c4418cbb2ad8deac315f3b; no voters:
I20250109 21:03:22.737946 17892 raft_consensus.cc:2798] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:22.738148 17892 raft_consensus.cc:695] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [term 1 LEADER]: Becoming Leader. State: Replica: a89c6dd645ff4d2da0d8dc7c51abd383, State: Running, Role: LEADER
I20250109 21:03:22.738447 17892 consensus_queue.cc:237] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556f3282fa904792b4f25319a95bccc8" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34843 } } peers { permanent_uuid: "cee2e84160c4418cbb2ad8deac315f3b" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44843 } } peers { permanent_uuid: "a89c6dd645ff4d2da0d8dc7c51abd383" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44639 } }
I20250109 21:03:22.741253 17413 catalog_manager.cc:5526] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 reported cstate change: term changed from 0 to 1, leader changed from <none> to a89c6dd645ff4d2da0d8dc7c51abd383 (127.12.126.67). New cstate: current_term: 1 leader_uuid: "a89c6dd645ff4d2da0d8dc7c51abd383" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556f3282fa904792b4f25319a95bccc8" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cee2e84160c4418cbb2ad8deac315f3b" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a89c6dd645ff4d2da0d8dc7c51abd383" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44639 } health_report { overall_health: HEALTHY } } }
W20250109 21:03:22.763651 17737 tablet.cc:2367] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:03:22.799567 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.796989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41210 (local address 127.12.126.67:44639)
0109 21:03:22.797117 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:22.797120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:22.797135 (+ 15us) server_negotiation.cc:407] Connection header received
0109 21:03:22.797213 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:22.797215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:22.797268 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:22.797357 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:22.797651 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.798122 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:22.798686 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:22.798829 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:22.798912 (+ 83us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:22.799297 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:22.799396 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:22.799437 (+ 41us) server_negotiation.cc:299] Negotiation successful
0109 21:03:22.799476 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250109 21:03:22.804811 17690 raft_consensus.cc:1270] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b [term 1 FOLLOWER]: Refusing update from remote peer a89c6dd645ff4d2da0d8dc7c51abd383: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:22.804811 17557 raft_consensus.cc:1270] T e32cd134bd6f47799f1c286b57d6cf6f P 556f3282fa904792b4f25319a95bccc8 [term 1 FOLLOWER]: Refusing update from remote peer a89c6dd645ff4d2da0d8dc7c51abd383: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:22.805423 17892 consensus_queue.cc:1035] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cee2e84160c4418cbb2ad8deac315f3b" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:22.805604 17896 consensus_queue.cc:1035] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [LEADER]: Connected to new peer: Peer: permanent_uuid: "556f3282fa904792b4f25319a95bccc8" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:22.813669 17901 mvcc.cc:204] Tried to move back new op lower bound from 7112526245082046464 to 7112526244818976768. Current Snapshot: MvccSnapshot[applied={T|T < 7112526245082046464}]
I20250109 21:03:22.819394 17412 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50928:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:22.819548 17412 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50928:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:22.821563 17412 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8968e9633a3c4b6e9809c98f12be40f0: Sending DeleteTablet for 3 replicas of tablet e32cd134bd6f47799f1c286b57d6cf6f
I20250109 21:03:22.822300 17670 tablet_service.cc:1514] Processing DeleteTablet for tablet e32cd134bd6f47799f1c286b57d6cf6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:22 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:49650
I20250109 21:03:22.822346 17803 tablet_service.cc:1514] Processing DeleteTablet for tablet e32cd134bd6f47799f1c286b57d6cf6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:22 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:41202
I20250109 21:03:22.822746 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 17475
I20250109 21:03:22.822949 17908 tablet_replica.cc:331] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b: stopping tablet replica
I20250109 21:03:22.822912 17537 tablet_service.cc:1514] Processing DeleteTablet for tablet e32cd134bd6f47799f1c286b57d6cf6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:22 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:46554
I20250109 21:03:22.823195 17908 raft_consensus.cc:2238] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:22.823509 17908 raft_consensus.cc:2267] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:22.823830 17909 tablet_replica.cc:331] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383: stopping tablet replica
I20250109 21:03:22.824895 17908 ts_tablet_manager.cc:1905] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:22.825300 17909 raft_consensus.cc:2238] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:03:22.825596 17909 raft_consensus.cc:2267] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:22.826894 17909 ts_tablet_manager.cc:1905] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:22.827947 17908 ts_tablet_manager.cc:1918] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:03:22.828090 17908 log.cc:1198] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/e32cd134bd6f47799f1c286b57d6cf6f
I20250109 21:03:22.828493 17908 ts_tablet_manager.cc:1939] T e32cd134bd6f47799f1c286b57d6cf6f P cee2e84160c4418cbb2ad8deac315f3b: Deleting consensus metadata
I20250109 21:03:22.829329 17909 ts_tablet_manager.cc:1918] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:03:22.829686 17909 log.cc:1198] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/e32cd134bd6f47799f1c286b57d6cf6f
I20250109 21:03:22.829669 17397 catalog_manager.cc:4872] TS cee2e84160c4418cbb2ad8deac315f3b (127.12.126.66:44843): tablet e32cd134bd6f47799f1c286b57d6cf6f (table test-table [id=842e5d9ef8f746fe9ca1ccbf9fe91cb1]) successfully deleted
I20250109 21:03:22.830291 17909 ts_tablet_manager.cc:1939] T e32cd134bd6f47799f1c286b57d6cf6f P a89c6dd645ff4d2da0d8dc7c51abd383: Deleting consensus metadata
W20250109 21:03:22.831046 17400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.126.65:34843 (error 108)
I20250109 21:03:22.831179 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 17608
I20250109 21:03:22.831701 17399 catalog_manager.cc:4872] TS a89c6dd645ff4d2da0d8dc7c51abd383 (127.12.126.67:44639): tablet e32cd134bd6f47799f1c286b57d6cf6f (table test-table [id=842e5d9ef8f746fe9ca1ccbf9fe91cb1]) successfully deleted
I20250109 21:03:22.832408 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:22.832073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:34843 (local address 127.0.0.1:46560)
0109 21:03:22.832262 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:22.832327 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.65:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20250109 21:03:22.832569 17400 catalog_manager.cc:4615] TS 556f3282fa904792b4f25319a95bccc8 (127.12.126.65:34843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e32cd134bd6f47799f1c286b57d6cf6f: Network error: Client connection negotiation failed: client connection to 127.12.126.65:34843: connect: Connection refused (error 111)
I20250109 21:03:22.837260 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 17741
I20250109 21:03:22.842653 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 17382
2025-01-09T21:03:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2335 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2025-01-09T21:03:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:03:22Z Disabled control of system clock
I20250109 21:03:22.871196 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:36927
--webserver_interface=127.12.126.126
--webserver_port=0
--builtin_ntp_servers=127.12.126.84:36807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:36927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20250109 21:03:22.972453 17914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:22.972678 17914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:22.972723 17914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:22.975724 17914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:22.975787 17914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:22.975803 17914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:22.975824 17914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:22.975847 17914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:03:22.979302 17914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:36807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:36927
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:36927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:22.980195 17914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:22.981199 17914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:22.988343 17922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:22.989256 17919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:22.989913 17920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:22.990854 17914 server_base.cc:1034] running on GCE node
I20250109 21:03:22.991398 17914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:22.992264 17914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:22.993409 17914 hybrid_clock.cc:648] HybridClock initialized: now 1736456602993393 us; error 29 us; skew 500 ppm
I20250109 21:03:22.995383 17914 webserver.cc:458] Webserver started at http://127.12.126.126:39889/ using document root <none> and password file <none>
I20250109 21:03:22.995942 17914 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:22.995997 17914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:22.996162 17914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:22.997915 17914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "3c919c23ed304d67873942841b2ff019"
format_stamp: "Formatted at 2025-01-09 21:03:22 on dist-test-slave-g0zb"
server_key: "a6ac582d555f3fbad487069d7fb79ccc"
server_key_iv: "54f29282fb971e62f21d3fa35fb01b95"
server_key_version: "encryptionkey@0"
I20250109 21:03:22.998405 17914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "3c919c23ed304d67873942841b2ff019"
format_stamp: "Formatted at 2025-01-09 21:03:22 on dist-test-slave-g0zb"
server_key: "a6ac582d555f3fbad487069d7fb79ccc"
server_key_iv: "54f29282fb971e62f21d3fa35fb01b95"
server_key_version: "encryptionkey@0"
I20250109 21:03:23.001510 17914 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:03:23.003603 17928 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:23.004570 17914 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250109 21:03:23.004675 17914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "3c919c23ed304d67873942841b2ff019"
format_stamp: "Formatted at 2025-01-09 21:03:22 on dist-test-slave-g0zb"
server_key: "a6ac582d555f3fbad487069d7fb79ccc"
server_key_iv: "54f29282fb971e62f21d3fa35fb01b95"
server_key_version: "encryptionkey@0"
I20250109 21:03:23.004761 17914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:23.009255 17914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:23.009778 17914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:23.009917 17914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:23.015800 17914 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:36927
I20250109 21:03:23.015810 17980 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:36927 every 8 connection(s)
I20250109 21:03:23.016655 17914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:03:23.019001 17981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:23.023975 17981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019: Bootstrap starting.
I20250109 21:03:23.025599 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 17914
I20250109 21:03:23.025719 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:03:23.025979 12793 external_mini_cluster.cc:1453] Setting key 8c8672077f751590fead2cb7559db6e6
I20250109 21:03:23.026430 17981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:23.027307 17981 log.cc:826] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:23.028851 17981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019: No bootstrap required, opened a new log
I20250109 21:03:23.030593 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.027141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41988 (local address 127.12.126.126:36927)
0109 21:03:23.027723 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:23.027731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:23.027762 (+ 31us) server_negotiation.cc:407] Connection header received
0109 21:03:23.028322 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:23.028339 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:23.028587 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:23.028888 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:23.029602 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:23.029607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:23.029615 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:03:23.029625 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:23.029666 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:03:23.029676 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:23.029771 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:23.029880 (+ 109us) server_negotiation.cc:299] Negotiation successful
0109 21:03:23.030073 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":108,"threads_started":1}
I20250109 21:03:23.031780 17981 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c919c23ed304d67873942841b2ff019" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 36927 } }
I20250109 21:03:23.031939 17981 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:23.031999 17981 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c919c23ed304d67873942841b2ff019, State: Initialized, Role: FOLLOWER
I20250109 21:03:23.032380 17981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c919c23ed304d67873942841b2ff019" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 36927 } }
I20250109 21:03:23.032518 17981 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:03:23.032577 17981 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:03:23.032672 17981 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:23.033389 17981 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c919c23ed304d67873942841b2ff019" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 36927 } }
I20250109 21:03:23.033663 17981 leader_election.cc:304] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c919c23ed304d67873942841b2ff019; no voters:
I20250109 21:03:23.033892 17981 leader_election.cc:290] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:03:23.033978 17986 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:23.034166 17986 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019 [term 1 LEADER]: Becoming Leader. State: Replica: 3c919c23ed304d67873942841b2ff019, State: Running, Role: LEADER
I20250109 21:03:23.034468 17986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c919c23ed304d67873942841b2ff019" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 36927 } }
I20250109 21:03:23.034893 17981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:23.036031 17988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c919c23ed304d67873942841b2ff019. Latest consensus state: current_term: 1 leader_uuid: "3c919c23ed304d67873942841b2ff019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c919c23ed304d67873942841b2ff019" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 36927 } } }
I20250109 21:03:23.036026 17987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c919c23ed304d67873942841b2ff019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c919c23ed304d67873942841b2ff019" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 36927 } } }
I20250109 21:03:23.036175 17988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:23.036183 17987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:23.036522 17995 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:03:23.038992 17995 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:03:23.044152 17995 catalog_manager.cc:1348] Generated new cluster ID: f8404685f91f4cef8d76d45a29a336d1
I20250109 21:03:23.044206 17995 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:03:23.053493 17995 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:03:23.053752 17995 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:03:23.058528 17995 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019: Generated new TSK 0
I20250109 21:03:23.059088 17995 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:03:23.062695 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:36927
--builtin_ntp_servers=127.12.126.84:36807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20250109 21:03:23.164287 18005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:23.164502 18005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:23.164592 18005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:23.167557 18005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:23.167605 18005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:23.167701 18005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:03:23.171124 18005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:36807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:36927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:23.172029 18005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:23.173089 18005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:23.181732 18010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:23.182533 18011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:23.183243 18013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:23.184062 18005 server_base.cc:1034] running on GCE node
I20250109 21:03:23.184399 18005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:23.184937 18005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:23.186098 18005 hybrid_clock.cc:648] HybridClock initialized: now 1736456603186082 us; error 39 us; skew 500 ppm
I20250109 21:03:23.188066 18005 webserver.cc:458] Webserver started at http://127.12.126.65:46881/ using document root <none> and password file <none>
I20250109 21:03:23.188632 18005 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:23.188689 18005 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:23.188860 18005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:23.190536 18005 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "e9f4e5829a6f4a209bf87a87b1115cdc"
format_stamp: "Formatted at 2025-01-09 21:03:23 on dist-test-slave-g0zb"
server_key: "707910eaa2deede15b1d35755909b76a"
server_key_iv: "64fc60967f75f527e377a6f78db1dfed"
server_key_version: "encryptionkey@0"
I20250109 21:03:23.191004 18005 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "e9f4e5829a6f4a209bf87a87b1115cdc"
format_stamp: "Formatted at 2025-01-09 21:03:23 on dist-test-slave-g0zb"
server_key: "707910eaa2deede15b1d35755909b76a"
server_key_iv: "64fc60967f75f527e377a6f78db1dfed"
server_key_version: "encryptionkey@0"
I20250109 21:03:23.194099 18005 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20250109 21:03:23.196214 18019 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:23.197145 18005 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:03:23.197242 18005 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "e9f4e5829a6f4a209bf87a87b1115cdc"
format_stamp: "Formatted at 2025-01-09 21:03:23 on dist-test-slave-g0zb"
server_key: "707910eaa2deede15b1d35755909b76a"
server_key_iv: "64fc60967f75f527e377a6f78db1dfed"
server_key_version: "encryptionkey@0"
I20250109 21:03:23.197335 18005 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:23.201910 18005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:23.202410 18005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:23.202555 18005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:23.203050 18005 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:23.203845 18005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:23.203892 18005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:23.203943 18005 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:23.203974 18005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:23.211899 18005 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:41059
I20250109 21:03:23.211920 18132 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:41059 every 8 connection(s)
I20250109 21:03:23.212702 18005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:03:23.217774 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 18005
I20250109 21:03:23.217893 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:03:23.218020 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.214167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:57711 (local address 127.12.126.126:36927)
0109 21:03:23.214303 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:23.214306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:23.215151 (+ 845us) server_negotiation.cc:407] Connection header received
0109 21:03:23.215877 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:23.215881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:23.215931 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:23.216014 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:23.217515 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:23.217519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:23.217522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:03:23.217535 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:23.217577 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:03:23.217587 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:23.217685 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:23.217826 (+ 141us) server_negotiation.cc:299] Negotiation successful
0109 21:03:23.217873 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250109 21:03:23.218189 12793 external_mini_cluster.cc:1453] Setting key 5a533ac088f4c7cb71371f5f73239d40
I20250109 21:03:23.218648 18135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.214421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:36927 (local address 127.12.126.65:57711)
0109 21:03:23.215033 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:23.215063 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:23.215689 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:23.216159 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:23.216166 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:23.216244 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:23.217263 (+ 1019us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:03:23.217265 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:23.217294 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:03:23.217347 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:23.217683 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:23.217690 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:23.217695 (+ 5us) client_negotiation.cc:769] Sending connection context
0109 21:03:23.217858 (+ 163us) client_negotiation.cc:240] Negotiation successful
0109 21:03:23.218031 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":108,"threads_started":1}
I20250109 21:03:23.219671 18133 heartbeater.cc:346] Connected to a master server at 127.12.126.126:36927
I20250109 21:03:23.219882 18133 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:23.219993 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:36927
--builtin_ntp_servers=127.12.126.84:36807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20250109 21:03:23.220270 18133 heartbeater.cc:510] Master 127.12.126.126:36927 requested a full tablet report, sending...
I20250109 21:03:23.221771 17944 ts_manager.cc:194] Registered new tserver with Master: e9f4e5829a6f4a209bf87a87b1115cdc (127.12.126.65:41059)
W20250109 21:03:23.320997 18136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:23.321214 18136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:23.321291 18136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:23.324638 18136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:23.324698 18136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:23.324819 18136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:03:23.328400 18136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:36807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:36927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:23.329376 18136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:23.330415 18136 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:23.338500 18141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:23.339313 18142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:23.340019 18144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:23.340873 18136 server_base.cc:1034] running on GCE node
I20250109 21:03:23.341193 18136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:23.341670 18136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:23.342834 18136 hybrid_clock.cc:648] HybridClock initialized: now 1736456603342819 us; error 26 us; skew 500 ppm
I20250109 21:03:23.344607 18136 webserver.cc:458] Webserver started at http://127.12.126.66:45507/ using document root <none> and password file <none>
I20250109 21:03:23.345125 18136 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:23.345185 18136 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:23.345362 18136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:23.347070 18136 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "802ba65b364b4da1ac32389cd85cfbd3"
format_stamp: "Formatted at 2025-01-09 21:03:23 on dist-test-slave-g0zb"
server_key: "f39527248038a296efd03bd10bb5e526"
server_key_iv: "01a06019343f2ce6d88847af7f68900c"
server_key_version: "encryptionkey@0"
I20250109 21:03:23.347505 18136 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "802ba65b364b4da1ac32389cd85cfbd3"
format_stamp: "Formatted at 2025-01-09 21:03:23 on dist-test-slave-g0zb"
server_key: "f39527248038a296efd03bd10bb5e526"
server_key_iv: "01a06019343f2ce6d88847af7f68900c"
server_key_version: "encryptionkey@0"
I20250109 21:03:23.350486 18136 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250109 21:03:23.352517 18150 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:23.353391 18136 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:03:23.353494 18136 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "802ba65b364b4da1ac32389cd85cfbd3"
format_stamp: "Formatted at 2025-01-09 21:03:23 on dist-test-slave-g0zb"
server_key: "f39527248038a296efd03bd10bb5e526"
server_key_iv: "01a06019343f2ce6d88847af7f68900c"
server_key_version: "encryptionkey@0"
I20250109 21:03:23.353574 18136 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:23.358119 18136 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:23.358568 18136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:23.358703 18136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:23.359165 18136 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:23.359930 18136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:23.359975 18136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:23.360030 18136 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:23.360057 18136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:23.368084 18136 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:39155
I20250109 21:03:23.368116 18263 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:39155 every 8 connection(s)
I20250109 21:03:23.368901 18136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:03:23.374174 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.370408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:38401 (local address 127.12.126.126:36927)
0109 21:03:23.370538 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:23.370542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:23.371359 (+ 817us) server_negotiation.cc:407] Connection header received
0109 21:03:23.372074 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:23.372077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:23.372129 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:23.372211 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:23.373705 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:23.373708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:23.373709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:03:23.373721 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:23.373741 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:03:23.373748 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:23.373832 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:23.373983 (+ 151us) server_negotiation.cc:299] Negotiation successful
0109 21:03:23.374030 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250109 21:03:23.374917 18266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.370636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:36927 (local address 127.12.126.66:38401)
0109 21:03:23.371221 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:23.371255 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:23.371873 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:23.372362 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:23.372370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:23.372444 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:23.373466 (+ 1022us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:03:23.373468 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:23.373497 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:03:23.373553 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:23.373820 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:23.373828 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:23.373833 (+ 5us) client_negotiation.cc:769] Sending connection context
0109 21:03:23.374031 (+ 198us) client_negotiation.cc:240] Negotiation successful
0109 21:03:23.374259 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":120,"threads_started":1}
I20250109 21:03:23.375382 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 18136
I20250109 21:03:23.375483 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:03:23.375780 12793 external_mini_cluster.cc:1453] Setting key d9bf0d0eaa1288bcc5fa11fb219fcf0c
I20250109 21:03:23.375914 18264 heartbeater.cc:346] Connected to a master server at 127.12.126.126:36927
I20250109 21:03:23.376130 18264 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:23.376549 18264 heartbeater.cc:510] Master 127.12.126.126:36927 requested a full tablet report, sending...
I20250109 21:03:23.377521 17944 ts_manager.cc:194] Registered new tserver with Master: 802ba65b364b4da1ac32389cd85cfbd3 (127.12.126.66:39155)
I20250109 21:03:23.377969 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:36927
--builtin_ntp_servers=127.12.126.84:36807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20250109 21:03:23.476034 18267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:23.476222 18267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:23.476297 18267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:23.479215 18267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:23.479259 18267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:23.479365 18267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:03:23.482857 18267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:36807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:36927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:23.483788 18267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:23.484826 18267 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:23.492663 18275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:23.493292 18273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:23.494089 18272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:23.494899 18267 server_base.cc:1034] running on GCE node
I20250109 21:03:23.495204 18267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:23.495688 18267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:23.496892 18267 hybrid_clock.cc:648] HybridClock initialized: now 1736456603496876 us; error 29 us; skew 500 ppm
I20250109 21:03:23.498605 18267 webserver.cc:458] Webserver started at http://127.12.126.67:34895/ using document root <none> and password file <none>
I20250109 21:03:23.499130 18267 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:23.499188 18267 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:23.499336 18267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:23.500926 18267 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "1e0a8a6377dd476a8386622e0ea84dc4"
format_stamp: "Formatted at 2025-01-09 21:03:23 on dist-test-slave-g0zb"
server_key: "9ba8e6c93b73c4e06b7d6ca4a5e85dca"
server_key_iv: "3ca09ab38c5a823f95a2e29b1ca817ef"
server_key_version: "encryptionkey@0"
I20250109 21:03:23.501420 18267 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "1e0a8a6377dd476a8386622e0ea84dc4"
format_stamp: "Formatted at 2025-01-09 21:03:23 on dist-test-slave-g0zb"
server_key: "9ba8e6c93b73c4e06b7d6ca4a5e85dca"
server_key_iv: "3ca09ab38c5a823f95a2e29b1ca817ef"
server_key_version: "encryptionkey@0"
I20250109 21:03:23.504536 18267 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:03:23.506498 18281 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:23.507470 18267 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:23.507571 18267 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "1e0a8a6377dd476a8386622e0ea84dc4"
format_stamp: "Formatted at 2025-01-09 21:03:23 on dist-test-slave-g0zb"
server_key: "9ba8e6c93b73c4e06b7d6ca4a5e85dca"
server_key_iv: "3ca09ab38c5a823f95a2e29b1ca817ef"
server_key_version: "encryptionkey@0"
I20250109 21:03:23.507651 18267 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:23.512102 18267 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:23.512547 18267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:23.512705 18267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:23.513124 18267 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:23.513906 18267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:23.513952 18267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:23.514007 18267 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:23.514036 18267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:23.521941 18267 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:34283
I20250109 21:03:23.521988 18394 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:34283 every 8 connection(s)
I20250109 21:03:23.522780 18267 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:03:23.527911 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.524292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:48097 (local address 127.12.126.126:36927)
0109 21:03:23.524433 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:23.524437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:23.525219 (+ 782us) server_negotiation.cc:407] Connection header received
0109 21:03:23.525940 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:23.525943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:23.525991 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:23.526070 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:23.527501 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:23.527505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:23.527508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:03:23.527519 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:23.527539 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:03:23.527546 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:23.527624 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:23.527743 (+ 119us) server_negotiation.cc:299] Negotiation successful
0109 21:03:23.527785 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250109 21:03:23.528543 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.524548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:36927 (local address 127.12.126.67:48097)
0109 21:03:23.525092 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:23.525125 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:23.525745 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:23.526219 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:23.526226 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:23.526297 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:23.527276 (+ 979us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:03:23.527278 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:23.527307 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:03:23.527358 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:23.527616 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:23.527622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:23.527628 (+ 6us) client_negotiation.cc:769] Sending connection context
0109 21:03:23.527794 (+ 166us) client_negotiation.cc:240] Negotiation successful
0109 21:03:23.527982 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":90,"threads_started":1}
I20250109 21:03:23.529414 18395 heartbeater.cc:346] Connected to a master server at 127.12.126.126:36927
I20250109 21:03:23.529598 18395 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:23.529923 18395 heartbeater.cc:510] Master 127.12.126.126:36927 requested a full tablet report, sending...
I20250109 21:03:23.530789 17944 ts_manager.cc:194] Registered new tserver with Master: 1e0a8a6377dd476a8386622e0ea84dc4 (127.12.126.67:34283)
I20250109 21:03:23.532608 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 18267
I20250109 21:03:23.532697 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:03:23.532891 12793 external_mini_cluster.cc:1453] Setting key b182cce31159eeca4157468e8fc277e0
I20250109 21:03:23.534541 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 21:03:23.537936 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.536337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41996 (local address 127.12.126.126:36927)
0109 21:03:23.536464 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:23.536468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:23.536683 (+ 215us) server_negotiation.cc:407] Connection header received
0109 21:03:23.536840 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:23.536843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:23.536875 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:23.536943 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:23.537599 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:23.537602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:23.537605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:03:23.537615 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:23.537633 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:03:23.537643 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:23.537715 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:23.537787 (+ 72us) server_negotiation.cc:299] Negotiation successful
0109 21:03:23.537821 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250109 21:03:23.540326 17944 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:03:23.542666 17944 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:03:23.552599 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.550106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:34283 (local address 127.0.0.1:52890)
0109 21:03:23.550629 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:23.550645 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:23.550714 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:23.551311 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:23.551316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:23.551332 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:23.551857 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:03:23.551859 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:23.551869 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:03:23.551888 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:23.552277 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:23.552287 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:23.552296 (+ 9us) client_negotiation.cc:769] Sending connection context
0109 21:03:23.552406 (+ 110us) client_negotiation.cc:240] Negotiation successful
0109 21:03:23.552469 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":68,"threads_started":1}
I20250109 21:03:23.552647 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.549487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:41059 (local address 127.0.0.1:52778)
0109 21:03:23.550030 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:23.550066 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:23.550155 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:23.551581 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:23.551583 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:23.551598 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:23.552034 (+ 436us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:03:23.552035 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:23.552042 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:03:23.552056 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:23.552366 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:23.552369 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:23.552374 (+ 5us) client_negotiation.cc:769] Sending connection context
0109 21:03:23.552462 (+ 88us) client_negotiation.cc:240] Negotiation successful
0109 21:03:23.552502 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":71,"threads_started":1}
I20250109 21:03:23.552850 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.549631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:39155 (local address 127.0.0.1:46498)
0109 21:03:23.550098 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:23.550114 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:23.550188 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:23.551048 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:23.551054 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:23.551112 (+ 58us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:23.551642 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:03:23.551646 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:23.551682 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:03:23.551744 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:23.552535 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:23.552537 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:23.552540 (+ 3us) client_negotiation.cc:769] Sending connection context
0109 21:03:23.552589 (+ 49us) client_negotiation.cc:240] Negotiation successful
0109 21:03:23.552614 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":62,"threads_started":1}
I20250109 21:03:23.552860 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.549618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52778 (local address 127.12.126.65:41059)
0109 21:03:23.550395 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:23.550399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:23.550422 (+ 23us) server_negotiation.cc:407] Connection header received
0109 21:03:23.550495 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:23.550502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:23.550922 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:23.551049 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:23.552150 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:23.552155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:23.552164 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:03:23.552194 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:23.552239 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:03:23.552253 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:23.552358 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:23.552486 (+ 128us) server_negotiation.cc:299] Negotiation successful
0109 21:03:23.552547 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":112,"threads_started":1}
I20250109 21:03:23.553045 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.550563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.12.126.67:34283)
0109 21:03:23.550945 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:23.550951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:23.550973 (+ 22us) server_negotiation.cc:407] Connection header received
0109 21:03:23.551028 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:23.551032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:23.551150 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:23.551518 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:23.552079 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:23.552086 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:23.552095 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:03:23.552106 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:23.552147 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:03:23.552159 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:23.552305 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:23.552828 (+ 523us) server_negotiation.cc:299] Negotiation successful
0109 21:03:23.552889 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":93,"threads_started":1}
I20250109 21:03:23.553539 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.549970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46498 (local address 127.12.126.66:39155)
0109 21:03:23.550347 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:23.550353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:23.550375 (+ 22us) server_negotiation.cc:407] Connection header received
0109 21:03:23.550433 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:23.550438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:23.550908 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:23.551046 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:23.552348 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:23.552353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:23.552360 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:03:23.552370 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:23.552411 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:03:23.552425 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:23.553286 (+ 861us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:23.553330 (+ 44us) server_negotiation.cc:299] Negotiation successful
0109 21:03:23.553385 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":112,"threads_started":1}
I20250109 21:03:23.555220 18329 tablet_service.cc:1467] Processing CreateTablet for tablet 23cc0ab76f59483da20948f23f4ffb17 (DEFAULT_TABLE table=test-table [id=85a419ccb80944faaaef2500a8de1171]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:23.555219 18067 tablet_service.cc:1467] Processing CreateTablet for tablet 23cc0ab76f59483da20948f23f4ffb17 (DEFAULT_TABLE table=test-table [id=85a419ccb80944faaaef2500a8de1171]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:23.555642 18198 tablet_service.cc:1467] Processing CreateTablet for tablet 23cc0ab76f59483da20948f23f4ffb17 (DEFAULT_TABLE table=test-table [id=85a419ccb80944faaaef2500a8de1171]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:23.555997 18329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23cc0ab76f59483da20948f23f4ffb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:23.556048 18067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23cc0ab76f59483da20948f23f4ffb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:23.556376 18198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23cc0ab76f59483da20948f23f4ffb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:23.561798 18409 tablet_bootstrap.cc:492] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4: Bootstrap starting.
I20250109 21:03:23.561820 18410 tablet_bootstrap.cc:492] T 23cc0ab76f59483da20948f23f4ffb17 P e9f4e5829a6f4a209bf87a87b1115cdc: Bootstrap starting.
I20250109 21:03:23.562583 18411 tablet_bootstrap.cc:492] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3: Bootstrap starting.
I20250109 21:03:23.563980 18409 tablet_bootstrap.cc:654] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:23.563980 18410 tablet_bootstrap.cc:654] T 23cc0ab76f59483da20948f23f4ffb17 P e9f4e5829a6f4a209bf87a87b1115cdc: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:23.564203 18411 tablet_bootstrap.cc:654] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:23.564797 18410 log.cc:826] T 23cc0ab76f59483da20948f23f4ffb17 P e9f4e5829a6f4a209bf87a87b1115cdc: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:23.564797 18409 log.cc:826] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:23.564824 18411 log.cc:826] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:23.566416 18411 tablet_bootstrap.cc:492] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3: No bootstrap required, opened a new log
I20250109 21:03:23.566440 18410 tablet_bootstrap.cc:492] T 23cc0ab76f59483da20948f23f4ffb17 P e9f4e5829a6f4a209bf87a87b1115cdc: No bootstrap required, opened a new log
I20250109 21:03:23.566578 18411 ts_tablet_manager.cc:1397] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250109 21:03:23.566587 18409 tablet_bootstrap.cc:492] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4: No bootstrap required, opened a new log
I20250109 21:03:23.566596 18410 ts_tablet_manager.cc:1397] T 23cc0ab76f59483da20948f23f4ffb17 P e9f4e5829a6f4a209bf87a87b1115cdc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:03:23.566712 18409 ts_tablet_manager.cc:1397] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20250109 21:03:23.569264 18411 raft_consensus.cc:357] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f4e5829a6f4a209bf87a87b1115cdc" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 41059 } } peers { permanent_uuid: "802ba65b364b4da1ac32389cd85cfbd3" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 39155 } } peers { permanent_uuid: "1e0a8a6377dd476a8386622e0ea84dc4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 34283 } }
I20250109 21:03:23.569432 18411 raft_consensus.cc:383] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:23.569520 18411 raft_consensus.cc:738] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 802ba65b364b4da1ac32389cd85cfbd3, State: Initialized, Role: FOLLOWER
I20250109 21:03:23.569908 18411 consensus_queue.cc:260] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f4e5829a6f4a209bf87a87b1115cdc" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 41059 } } peers { permanent_uuid: "802ba65b364b4da1ac32389cd85cfbd3" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 39155 } } peers { permanent_uuid: "1e0a8a6377dd476a8386622e0ea84dc4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 34283 } }
I20250109 21:03:23.570248 18409 raft_consensus.cc:357] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f4e5829a6f4a209bf87a87b1115cdc" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 41059 } } peers { permanent_uuid: "802ba65b364b4da1ac32389cd85cfbd3" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 39155 } } peers { permanent_uuid: "1e0a8a6377dd476a8386622e0ea84dc4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 34283 } }
I20250109 21:03:23.570245 18410 raft_consensus.cc:357] T 23cc0ab76f59483da20948f23f4ffb17 P e9f4e5829a6f4a209bf87a87b1115cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f4e5829a6f4a209bf87a87b1115cdc" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 41059 } } peers { permanent_uuid: "802ba65b364b4da1ac32389cd85cfbd3" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 39155 } } peers { permanent_uuid: "1e0a8a6377dd476a8386622e0ea84dc4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 34283 } }
I20250109 21:03:23.570434 18409 raft_consensus.cc:383] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:23.570434 18410 raft_consensus.cc:383] T 23cc0ab76f59483da20948f23f4ffb17 P e9f4e5829a6f4a209bf87a87b1115cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:23.570473 18264 heartbeater.cc:502] Master 127.12.126.126:36927 was elected leader, sending a full tablet report...
I20250109 21:03:23.570503 18410 raft_consensus.cc:738] T 23cc0ab76f59483da20948f23f4ffb17 P e9f4e5829a6f4a209bf87a87b1115cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f4e5829a6f4a209bf87a87b1115cdc, State: Initialized, Role: FOLLOWER
I20250109 21:03:23.570536 18409 raft_consensus.cc:738] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e0a8a6377dd476a8386622e0ea84dc4, State: Initialized, Role: FOLLOWER
I20250109 21:03:23.570905 18409 consensus_queue.cc:260] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f4e5829a6f4a209bf87a87b1115cdc" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 41059 } } peers { permanent_uuid: "802ba65b364b4da1ac32389cd85cfbd3" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 39155 } } peers { permanent_uuid: "1e0a8a6377dd476a8386622e0ea84dc4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 34283 } }
I20250109 21:03:23.570905 18410 consensus_queue.cc:260] T 23cc0ab76f59483da20948f23f4ffb17 P e9f4e5829a6f4a209bf87a87b1115cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f4e5829a6f4a209bf87a87b1115cdc" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 41059 } } peers { permanent_uuid: "802ba65b364b4da1ac32389cd85cfbd3" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 39155 } } peers { permanent_uuid: "1e0a8a6377dd476a8386622e0ea84dc4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 34283 } }
I20250109 21:03:23.571683 18410 ts_tablet_manager.cc:1428] T 23cc0ab76f59483da20948f23f4ffb17 P e9f4e5829a6f4a209bf87a87b1115cdc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:03:23.571683 18409 ts_tablet_manager.cc:1428] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:03:23.571990 18395 heartbeater.cc:502] Master 127.12.126.126:36927 was elected leader, sending a full tablet report...
I20250109 21:03:23.571990 18133 heartbeater.cc:502] Master 127.12.126.126:36927 was elected leader, sending a full tablet report...
I20250109 21:03:23.573244 18411 ts_tablet_manager.cc:1428] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20250109 21:03:23.619688 18265 tablet.cc:2367] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:03:23.666246 18415 raft_consensus.cc:491] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:03:23.666450 18415 raft_consensus.cc:513] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f4e5829a6f4a209bf87a87b1115cdc" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 41059 } } peers { permanent_uuid: "802ba65b364b4da1ac32389cd85cfbd3" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 39155 } } peers { permanent_uuid: "1e0a8a6377dd476a8386622e0ea84dc4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 34283 } }
I20250109 21:03:23.667613 18415 leader_election.cc:290] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f4e5829a6f4a209bf87a87b1115cdc (127.12.126.65:41059), 1e0a8a6377dd476a8386622e0ea84dc4 (127.12.126.67:34283)
I20250109 21:03:23.669606 18266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.667784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:41059 (local address 127.12.126.66:50063)
0109 21:03:23.667935 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:23.667952 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:23.668025 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:23.668376 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:23.668380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:23.668394 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:23.669039 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:03:23.669041 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:23.669051 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:03:23.669074 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:23.669373 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:23.669379 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:23.669384 (+ 5us) client_negotiation.cc:769] Sending connection context
0109 21:03:23.669464 (+ 80us) client_negotiation.cc:240] Negotiation successful
0109 21:03:23.669495 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20250109 21:03:23.669670 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.667898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:50063 (local address 127.12.126.65:41059)
0109 21:03:23.668036 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:23.668042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:23.668091 (+ 49us) server_negotiation.cc:407] Connection header received
0109 21:03:23.668139 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:23.668142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:23.668190 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:23.668262 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:23.669227 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:23.669232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:23.669235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:03:23.669248 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:23.669274 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:03:23.669287 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:23.669376 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:23.669475 (+ 99us) server_negotiation.cc:299] Negotiation successful
0109 21:03:23.669520 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250109 21:03:23.669865 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.668247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:33075 (local address 127.12.126.67:34283)
0109 21:03:23.668378 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:23.668382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:23.668527 (+ 145us) server_negotiation.cc:407] Connection header received
0109 21:03:23.668648 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:23.668652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:23.668697 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:23.668780 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:23.669470 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:23.669474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:23.669476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:03:23.669490 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:23.669513 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:03:23.669522 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:23.669615 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:23.669695 (+ 80us) server_negotiation.cc:299] Negotiation successful
0109 21:03:23.669739 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20250109 21:03:23.669898 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.668068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:34283 (local address 127.12.126.66:33075)
0109 21:03:23.668440 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:23.668455 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:23.668545 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:23.668806 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:23.668811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:23.668827 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:23.669313 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:03:23.669314 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:23.669322 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:03:23.669342 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:23.669603 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:23.669606 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:23.669609 (+ 3us) client_negotiation.cc:769] Sending connection context
0109 21:03:23.669703 (+ 94us) client_negotiation.cc:240] Negotiation successful
0109 21:03:23.669738 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":127,"threads_started":1}
I20250109 21:03:23.670449 18349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23cc0ab76f59483da20948f23f4ffb17" candidate_uuid: "802ba65b364b4da1ac32389cd85cfbd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e0a8a6377dd476a8386622e0ea84dc4" is_pre_election: true
I20250109 21:03:23.670452 18087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23cc0ab76f59483da20948f23f4ffb17" candidate_uuid: "802ba65b364b4da1ac32389cd85cfbd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f4e5829a6f4a209bf87a87b1115cdc" is_pre_election: true
I20250109 21:03:23.670719 18087 raft_consensus.cc:2463] T 23cc0ab76f59483da20948f23f4ffb17 P e9f4e5829a6f4a209bf87a87b1115cdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 802ba65b364b4da1ac32389cd85cfbd3 in term 0.
I20250109 21:03:23.670722 18349 raft_consensus.cc:2463] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 802ba65b364b4da1ac32389cd85cfbd3 in term 0.
I20250109 21:03:23.671139 18154 leader_election.cc:304] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e0a8a6377dd476a8386622e0ea84dc4, 802ba65b364b4da1ac32389cd85cfbd3; no voters:
I20250109 21:03:23.671366 18415 raft_consensus.cc:2798] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:03:23.671447 18415 raft_consensus.cc:491] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:03:23.671489 18415 raft_consensus.cc:3054] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:23.672451 18415 raft_consensus.cc:513] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f4e5829a6f4a209bf87a87b1115cdc" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 41059 } } peers { permanent_uuid: "802ba65b364b4da1ac32389cd85cfbd3" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 39155 } } peers { permanent_uuid: "1e0a8a6377dd476a8386622e0ea84dc4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 34283 } }
I20250109 21:03:23.672776 18415 leader_election.cc:290] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [CANDIDATE]: Term 1 election: Requested vote from peers e9f4e5829a6f4a209bf87a87b1115cdc (127.12.126.65:41059), 1e0a8a6377dd476a8386622e0ea84dc4 (127.12.126.67:34283)
I20250109 21:03:23.673132 18349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23cc0ab76f59483da20948f23f4ffb17" candidate_uuid: "802ba65b364b4da1ac32389cd85cfbd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e0a8a6377dd476a8386622e0ea84dc4"
I20250109 21:03:23.673260 18349 raft_consensus.cc:3054] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:23.673570 18087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23cc0ab76f59483da20948f23f4ffb17" candidate_uuid: "802ba65b364b4da1ac32389cd85cfbd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f4e5829a6f4a209bf87a87b1115cdc"
I20250109 21:03:23.673671 18087 raft_consensus.cc:3054] T 23cc0ab76f59483da20948f23f4ffb17 P e9f4e5829a6f4a209bf87a87b1115cdc [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:23.674213 18349 raft_consensus.cc:2463] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 802ba65b364b4da1ac32389cd85cfbd3 in term 1.
I20250109 21:03:23.674487 18154 leader_election.cc:304] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e0a8a6377dd476a8386622e0ea84dc4, 802ba65b364b4da1ac32389cd85cfbd3; no voters:
I20250109 21:03:23.674651 18087 raft_consensus.cc:2463] T 23cc0ab76f59483da20948f23f4ffb17 P e9f4e5829a6f4a209bf87a87b1115cdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 802ba65b364b4da1ac32389cd85cfbd3 in term 1.
I20250109 21:03:23.674629 18415 raft_consensus.cc:2798] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:23.674983 18415 raft_consensus.cc:695] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [term 1 LEADER]: Becoming Leader. State: Replica: 802ba65b364b4da1ac32389cd85cfbd3, State: Running, Role: LEADER
I20250109 21:03:23.675307 18415 consensus_queue.cc:237] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f4e5829a6f4a209bf87a87b1115cdc" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 41059 } } peers { permanent_uuid: "802ba65b364b4da1ac32389cd85cfbd3" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 39155 } } peers { permanent_uuid: "1e0a8a6377dd476a8386622e0ea84dc4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 34283 } }
I20250109 21:03:23.678066 17944 catalog_manager.cc:5526] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 802ba65b364b4da1ac32389cd85cfbd3 (127.12.126.66). New cstate: current_term: 1 leader_uuid: "802ba65b364b4da1ac32389cd85cfbd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f4e5829a6f4a209bf87a87b1115cdc" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 41059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "802ba65b364b4da1ac32389cd85cfbd3" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e0a8a6377dd476a8386622e0ea84dc4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 34283 } health_report { overall_health: UNKNOWN } } }
I20250109 21:03:23.701987 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.700604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.12.126.66:39155)
0109 21:03:23.700719 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:23.700723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:23.700737 (+ 14us) server_negotiation.cc:407] Connection header received
0109 21:03:23.700848 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:23.700851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:23.700902 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:23.700974 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:23.701655 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:23.701658 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:23.701660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:03:23.701670 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:23.701689 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:03:23.701696 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:23.701769 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:23.701822 (+ 53us) server_negotiation.cc:299] Negotiation successful
0109 21:03:23.701851 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250109 21:03:23.707082 18349 raft_consensus.cc:1270] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4 [term 1 FOLLOWER]: Refusing update from remote peer 802ba65b364b4da1ac32389cd85cfbd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:23.707083 18087 raft_consensus.cc:1270] T 23cc0ab76f59483da20948f23f4ffb17 P e9f4e5829a6f4a209bf87a87b1115cdc [term 1 FOLLOWER]: Refusing update from remote peer 802ba65b364b4da1ac32389cd85cfbd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:23.707648 18415 consensus_queue.cc:1035] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f4e5829a6f4a209bf87a87b1115cdc" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 41059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:23.707803 18419 consensus_queue.cc:1035] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e0a8a6377dd476a8386622e0ea84dc4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 34283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 21:03:23.713631 18134 tablet.cc:2367] T 23cc0ab76f59483da20948f23f4ffb17 P e9f4e5829a6f4a209bf87a87b1115cdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:03:23.722451 17945 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41996:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:23.722589 17945 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41996:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:23.724684 17945 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 3c919c23ed304d67873942841b2ff019: Sending DeleteTablet for 3 replicas of tablet 23cc0ab76f59483da20948f23f4ffb17
I20250109 21:03:23.725363 18067 tablet_service.cc:1514] Processing DeleteTablet for tablet 23cc0ab76f59483da20948f23f4ffb17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:23 UTC) from {username='slave'} at 127.0.0.1:52778
I20250109 21:03:23.725360 18198 tablet_service.cc:1514] Processing DeleteTablet for tablet 23cc0ab76f59483da20948f23f4ffb17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:23 UTC) from {username='slave'} at 127.0.0.1:46498
I20250109 21:03:23.725479 18329 tablet_service.cc:1514] Processing DeleteTablet for tablet 23cc0ab76f59483da20948f23f4ffb17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:23 UTC) from {username='slave'} at 127.0.0.1:52890
I20250109 21:03:23.725780 18432 tablet_replica.cc:331] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3: stopping tablet replica
I20250109 21:03:23.725859 18431 tablet_replica.cc:331] T 23cc0ab76f59483da20948f23f4ffb17 P e9f4e5829a6f4a209bf87a87b1115cdc: stopping tablet replica
I20250109 21:03:23.726063 18431 raft_consensus.cc:2238] T 23cc0ab76f59483da20948f23f4ffb17 P e9f4e5829a6f4a209bf87a87b1115cdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:23.726064 18432 raft_consensus.cc:2238] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:03:23.726356 18432 raft_consensus.cc:2267] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:23.726397 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 18005
I20250109 21:03:23.726423 18431 raft_consensus.cc:2267] T 23cc0ab76f59483da20948f23f4ffb17 P e9f4e5829a6f4a209bf87a87b1115cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:23.727222 18433 tablet_replica.cc:331] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4: stopping tablet replica
I20250109 21:03:23.727519 18433 raft_consensus.cc:2238] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:23.727756 18432 ts_tablet_manager.cc:1905] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:23.727830 18433 raft_consensus.cc:2267] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:23.729086 18433 ts_tablet_manager.cc:1905] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:23.730324 18432 ts_tablet_manager.cc:1918] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:03:23.730484 18432 log.cc:1198] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/23cc0ab76f59483da20948f23f4ffb17
I20250109 21:03:23.730937 18432 ts_tablet_manager.cc:1939] T 23cc0ab76f59483da20948f23f4ffb17 P 802ba65b364b4da1ac32389cd85cfbd3: Deleting consensus metadata
I20250109 21:03:23.731603 18433 ts_tablet_manager.cc:1918] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:03:23.731760 18433 log.cc:1198] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/23cc0ab76f59483da20948f23f4ffb17
I20250109 21:03:23.732082 17931 catalog_manager.cc:4872] TS 802ba65b364b4da1ac32389cd85cfbd3 (127.12.126.66:39155): tablet 23cc0ab76f59483da20948f23f4ffb17 (table test-table [id=85a419ccb80944faaaef2500a8de1171]) successfully deleted
I20250109 21:03:23.732144 18433 ts_tablet_manager.cc:1939] T 23cc0ab76f59483da20948f23f4ffb17 P 1e0a8a6377dd476a8386622e0ea84dc4: Deleting consensus metadata
I20250109 21:03:23.733026 17932 catalog_manager.cc:4872] TS 1e0a8a6377dd476a8386622e0ea84dc4 (127.12.126.67:34283): tablet 23cc0ab76f59483da20948f23f4ffb17 (table test-table [id=85a419ccb80944faaaef2500a8de1171]) successfully deleted
W20250109 21:03:23.736080 17932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.126.65:41059 (error 108)
I20250109 21:03:23.736330 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 18136
I20250109 21:03:23.737962 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.737648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:41059 (local address 127.0.0.1:41028)
0109 21:03:23.737793 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:23.737865 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.65:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20250109 21:03:23.738090 17932 catalog_manager.cc:4615] TS e9f4e5829a6f4a209bf87a87b1115cdc (127.12.126.65:41059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 23cc0ab76f59483da20948f23f4ffb17: Network error: Client connection negotiation failed: client connection to 127.12.126.65:41059: connect: Connection refused (error 111)
I20250109 21:03:23.741904 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 18267
I20250109 21:03:23.747493 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 17914
2025-01-09T21:03:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (904 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20250109 21:03:23.768183 12793 mini_oidc.cc:150] Starting JWKS server
I20250109 21:03:23.769496 12793 webserver.cc:458] Webserver started at https://127.0.0.1:43861/ using document root <none> and password file <none>
I20250109 21:03:23.769630 12793 mini_oidc.cc:186] Starting OIDC Discovery server
I20250109 21:03:23.769865 12793 webserver.cc:458] Webserver started at http://127.0.0.1:40853/ using document root <none> and password file <none>
2025-01-09T21:03:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:03:23Z Disabled control of system clock
I20250109 21:03:23.777261 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:37049
--webserver_interface=127.12.126.126
--webserver_port=0
--builtin_ntp_servers=127.12.126.84:46207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:37049
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43861/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/testchainca.cert with env {}
W20250109 21:03:23.877318 18440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:23.877519 18440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:23.877601 18440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:23.880591 18440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:23.880646 18440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:23.880672 18440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:23.880710 18440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:23.880746 18440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250109 21:03:23.880790 18440 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250109 21:03:23.880822 18440 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43861/jwks.json
I20250109 21:03:23.885000 18440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:46207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:37049
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:37049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43861/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:23.885944 18440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:23.886696 18440 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:23.894073 18448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:23.894740 18445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:23.895757 18446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:23.896530 18440 server_base.cc:1034] running on GCE node
I20250109 21:03:23.896983 18440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:23.897837 18440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:23.898980 18440 hybrid_clock.cc:648] HybridClock initialized: now 1736456603898969 us; error 29 us; skew 500 ppm
I20250109 21:03:23.900686 18440 webserver.cc:458] Webserver started at http://127.12.126.126:45215/ using document root <none> and password file <none>
I20250109 21:03:23.901213 18440 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:23.901294 18440 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:23.901463 18440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:23.903231 18440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "e585388b2aa749d492bd46c8425a473c"
format_stamp: "Formatted at 2025-01-09 21:03:23 on dist-test-slave-g0zb"
server_key: "ce94de21a6e393acdff48b86d01eef1d"
server_key_iv: "881416469821f08f73c88e87555b76de"
server_key_version: "encryptionkey@0"
I20250109 21:03:23.903693 18440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "e585388b2aa749d492bd46c8425a473c"
format_stamp: "Formatted at 2025-01-09 21:03:23 on dist-test-slave-g0zb"
server_key: "ce94de21a6e393acdff48b86d01eef1d"
server_key_iv: "881416469821f08f73c88e87555b76de"
server_key_version: "encryptionkey@0"
I20250109 21:03:23.906963 18440 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:03:23.908964 18454 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:23.909914 18440 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:23.910030 18440 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "e585388b2aa749d492bd46c8425a473c"
format_stamp: "Formatted at 2025-01-09 21:03:23 on dist-test-slave-g0zb"
server_key: "ce94de21a6e393acdff48b86d01eef1d"
server_key_iv: "881416469821f08f73c88e87555b76de"
server_key_version: "encryptionkey@0"
I20250109 21:03:23.910106 18440 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:23.930619 18440 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:23.931241 18440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:23.931398 18440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:23.937686 18440 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:37049
I20250109 21:03:23.937703 18506 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:37049 every 8 connection(s)
I20250109 21:03:23.938597 18440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:03:23.941013 18507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:23.941993 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 18440
I20250109 21:03:23.942111 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:03:23.942354 12793 external_mini_cluster.cc:1453] Setting key e4bef40b8cc9b986f5dea1acfa34c537
I20250109 21:03:23.946651 18507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c: Bootstrap starting.
I20250109 21:03:23.948855 18507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:23.949599 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:23.943502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.12.126.126:37049)
0109 21:03:23.944109 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:23.944117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:23.944149 (+ 32us) server_negotiation.cc:407] Connection header received
0109 21:03:23.944665 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:23.944681 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:23.944992 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:23.945309 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:23.945768 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:23.946529 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:23.947285 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:23.947553 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:23.948114 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:23.948150 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:23.948160 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:03:23.948507 (+ 347us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:23.948545 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:03:23.948561 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:23.948657 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:23.948810 (+ 153us) server_negotiation.cc:299] Negotiation successful
0109 21:03:23.949041 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":130,"threads_started":1}
I20250109 21:03:23.949607 18507 log.cc:826] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:23.951557 18507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c: No bootstrap required, opened a new log
I20250109 21:03:23.952287 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:03:23.954864 18507 raft_consensus.cc:357] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e585388b2aa749d492bd46c8425a473c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 37049 } }
I20250109 21:03:23.955050 18507 raft_consensus.cc:383] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:23.955108 18507 raft_consensus.cc:738] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e585388b2aa749d492bd46c8425a473c, State: Initialized, Role: FOLLOWER
I20250109 21:03:23.955509 18507 consensus_queue.cc:260] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e585388b2aa749d492bd46c8425a473c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 37049 } }
I20250109 21:03:23.955626 18507 raft_consensus.cc:397] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:03:23.955683 18507 raft_consensus.cc:491] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:03:23.955788 18507 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:23.956579 18507 raft_consensus.cc:513] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e585388b2aa749d492bd46c8425a473c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 37049 } }
I20250109 21:03:23.956889 18507 leader_election.cc:304] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e585388b2aa749d492bd46c8425a473c; no voters:
I20250109 21:03:23.957163 18507 leader_election.cc:290] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:03:23.957257 18512 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:23.957499 18512 raft_consensus.cc:695] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c [term 1 LEADER]: Becoming Leader. State: Replica: e585388b2aa749d492bd46c8425a473c, State: Running, Role: LEADER
I20250109 21:03:23.957832 18512 consensus_queue.cc:237] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e585388b2aa749d492bd46c8425a473c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 37049 } }
I20250109 21:03:23.958281 18507 sys_catalog.cc:564] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:23.958843 18512 sys_catalog.cc:455] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e585388b2aa749d492bd46c8425a473c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e585388b2aa749d492bd46c8425a473c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 37049 } } }
I20250109 21:03:23.959017 18512 sys_catalog.cc:458] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:23.959298 18513 sys_catalog.cc:455] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e585388b2aa749d492bd46c8425a473c. Latest consensus state: current_term: 1 leader_uuid: "e585388b2aa749d492bd46c8425a473c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e585388b2aa749d492bd46c8425a473c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 37049 } } }
I20250109 21:03:23.959439 18513 sys_catalog.cc:458] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:23.959462 18520 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:03:23.961975 18520 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:03:23.967082 18520 catalog_manager.cc:1348] Generated new cluster ID: 6d3192438bd743ccb404a919ef132231
I20250109 21:03:23.967134 18520 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:03:23.980402 18520 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:03:23.981268 18520 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:03:23.991499 18520 catalog_manager.cc:5899] T 00000000000000000000000000000000 P e585388b2aa749d492bd46c8425a473c: Generated new TSK 0
I20250109 21:03:23.992198 18520 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:03:24.025755 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:24.003858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.12.126.126:37049)
0109 21:03:24.003979 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:24.003983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:24.004116 (+ 133us) server_negotiation.cc:407] Connection header received
0109 21:03:24.004352 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:24.004354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:24.004402 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:24.004504 (+ 102us) server_negotiation.cc:226] Negotiated authn=JWT
0109 21:03:24.004999 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:24.005543 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:24.006449 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:24.006590 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:24.006673 (+ 83us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0109 21:03:24.024983 (+ 18310us) server_negotiation.cc:377] Sending JWT_EXCHANGE NegotiatePB response
0109 21:03:24.025077 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:24.025320 (+ 243us) server_negotiation.cc:299] Negotiation successful
0109 21:03:24.025376 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48,"thread_start_us":127,"threads_started":1}
I20250109 21:03:24.036827 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:24.033302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.12.126.126:37049)
0109 21:03:24.033412 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:24.033415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:24.033625 (+ 210us) server_negotiation.cc:407] Connection header received
0109 21:03:24.033758 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:24.033762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:24.033808 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:24.033874 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0109 21:03:24.034199 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:24.034683 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:24.035326 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:24.035507 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:24.035551 (+ 44us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0109 21:03:24.036487 (+ 936us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0109 21:03:24.036690 (+ 203us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37500: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":40}
W20250109 21:03:24.036900 18510 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37500: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20250109 21:03:24.037477 18543 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:24.033244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:37049 (local address 127.0.0.1:37500)
0109 21:03:24.033540 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:24.033553 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:24.033661 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:24.033890 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:24.033893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:24.033901 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0109 21:03:24.034090 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:24.034097 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:24.034788 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:24.034792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:24.035218 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:24.035227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:24.035330 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:24.035353 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0109 21:03:24.036658 (+ 1305us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0109 21:03:24.036800 (+ 142us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.126.126:37049: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":92,"threads_started":1}
I20250109 21:03:39.044567 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:39.041369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42832 (local address 127.12.126.126:37049)
0109 21:03:39.041649 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:39.041652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:39.041667 (+ 15us) server_negotiation.cc:407] Connection header received
0109 21:03:39.041726 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:39.041728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:39.041772 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:39.041858 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0109 21:03:39.042159 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.042697 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:39.043365 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.043490 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:39.043530 (+ 40us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0109 21:03:39.044326 (+ 796us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0109 21:03:39.044463 (+ 137us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42832: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":88,"threads_started":1}
W20250109 21:03:39.044637 18549 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42832: JWT verification failed: token expired
W20250109 21:03:39.044704 18548 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:39.041233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:37049 (local address 127.0.0.1:42832)
0109 21:03:39.041532 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:39.041551 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:39.041647 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:39.041872 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:39.041874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:39.041880 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0109 21:03:39.042079 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:39.042083 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.042840 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:39.042844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:39.043265 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:39.043274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.043376 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:39.043395 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0109 21:03:39.044453 (+ 1058us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0109 21:03:39.044570 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.126.126:37049: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":98,"threads_started":1}
W20250109 21:03:39.047269 18549 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:39.046482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.12.126.126:37049)
0109 21:03:39.046624 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:39.046630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:39.046691 (+ 61us) server_negotiation.cc:407] Connection header received
0109 21:03:39.046845 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:39.046849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:39.046893 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:39.046970 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:39.047163 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42840: BlockingRecv error: recv got EOF from 127.0.0.1:42840 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20250109 21:03:39.050459 18559 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.12.126.126:37049
W20250109 21:03:39.051107 18549 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:39.050202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42850 (local address 127.12.126.126:37049)
0109 21:03:39.050302 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:39.050305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:39.050417 (+ 112us) server_negotiation.cc:407] Connection header received
0109 21:03:39.050604 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:39.050609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:39.050655 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:39.050740 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:39.051012 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42850: BlockingRecv error: recv got EOF from 127.0.0.1:42850 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20250109 21:03:39.051296 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 18440
2025-01-09T21:03:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15323 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20250109 21:03:39.091603 12793 mini_oidc.cc:150] Starting JWKS server
I20250109 21:03:39.092486 12793 webserver.cc:458] Webserver started at https://127.0.0.1:41675/ using document root <none> and password file <none>
I20250109 21:03:39.092582 12793 mini_oidc.cc:186] Starting OIDC Discovery server
I20250109 21:03:39.092801 12793 webserver.cc:458] Webserver started at http://127.0.0.1:39021/ using document root <none> and password file <none>
2025-01-09T21:03:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:03:39Z Disabled control of system clock
I20250109 21:03:39.100183 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:46531
--webserver_interface=127.12.126.126
--webserver_port=0
--builtin_ntp_servers=127.12.126.84:42313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:46531
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41675/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/testchainca.cert with env {}
W20250109 21:03:39.200709 18565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:39.200927 18565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:39.200975 18565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:39.204012 18565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:39.204066 18565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:39.204082 18565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:39.204100 18565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:39.204123 18565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250109 21:03:39.204141 18565 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250109 21:03:39.204160 18565 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41675/jwks.json
I20250109 21:03:39.208278 18565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:46531
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:46531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41675/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:39.209255 18565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:39.210009 18565 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:39.217540 18571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:39.218380 18570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:39.219132 18573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:39.220050 18565 server_base.cc:1034] running on GCE node
I20250109 21:03:39.220521 18565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:39.221408 18565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:39.222586 18565 hybrid_clock.cc:648] HybridClock initialized: now 1736456619222560 us; error 29 us; skew 500 ppm
I20250109 21:03:39.224522 18565 webserver.cc:458] Webserver started at http://127.12.126.126:45975/ using document root <none> and password file <none>
I20250109 21:03:39.225066 18565 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:39.225126 18565 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:39.225289 18565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:39.227128 18565 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "8dcd0be0c3a94b5197ec871677a5ad9b"
format_stamp: "Formatted at 2025-01-09 21:03:39 on dist-test-slave-g0zb"
server_key: "1182cf87991618508b5dfdfd7b749e8a"
server_key_iv: "31b4a12c3c58e8fa8251465625f71ee7"
server_key_version: "encryptionkey@0"
I20250109 21:03:39.227607 18565 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "8dcd0be0c3a94b5197ec871677a5ad9b"
format_stamp: "Formatted at 2025-01-09 21:03:39 on dist-test-slave-g0zb"
server_key: "1182cf87991618508b5dfdfd7b749e8a"
server_key_iv: "31b4a12c3c58e8fa8251465625f71ee7"
server_key_version: "encryptionkey@0"
I20250109 21:03:39.230861 18565 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250109 21:03:39.232942 18579 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:39.233894 18565 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250109 21:03:39.234020 18565 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "8dcd0be0c3a94b5197ec871677a5ad9b"
format_stamp: "Formatted at 2025-01-09 21:03:39 on dist-test-slave-g0zb"
server_key: "1182cf87991618508b5dfdfd7b749e8a"
server_key_iv: "31b4a12c3c58e8fa8251465625f71ee7"
server_key_version: "encryptionkey@0"
I20250109 21:03:39.234102 18565 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:39.243796 18565 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:39.244366 18565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:39.244534 18565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:39.250480 18565 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:46531
I20250109 21:03:39.250722 18631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:46531 every 8 connection(s)
I20250109 21:03:39.251463 18565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:03:39.253938 18632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:39.254869 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 18565
I20250109 21:03:39.254985 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:03:39.255280 12793 external_mini_cluster.cc:1453] Setting key 3ba8e5adb33c327aa177d7d7515eb4a0
I20250109 21:03:39.259299 18632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b: Bootstrap starting.
I20250109 21:03:39.261384 18632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:39.262099 18632 log.cc:826] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:39.262580 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:39.256517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38794 (local address 127.12.126.126:46531)
0109 21:03:39.257097 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:39.257106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:39.257138 (+ 32us) server_negotiation.cc:407] Connection header received
0109 21:03:39.257675 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:39.257691 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:39.258024 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:39.258339 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:39.258828 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.259636 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:39.260282 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.260571 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:39.261067 (+ 496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:39.261085 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:39.261095 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:03:39.261466 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:39.261511 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:03:39.261529 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:39.261617 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:39.261752 (+ 135us) server_negotiation.cc:299] Negotiation successful
0109 21:03:39.261949 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":116,"threads_started":1}
I20250109 21:03:39.263957 18632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b: No bootstrap required, opened a new log
I20250109 21:03:39.265060 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:03:39.267179 18632 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dcd0be0c3a94b5197ec871677a5ad9b" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 46531 } }
I20250109 21:03:39.267349 18632 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:39.267400 18632 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dcd0be0c3a94b5197ec871677a5ad9b, State: Initialized, Role: FOLLOWER
I20250109 21:03:39.267742 18632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dcd0be0c3a94b5197ec871677a5ad9b" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 46531 } }
I20250109 21:03:39.267838 18632 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:03:39.267879 18632 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:03:39.267951 18632 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:39.268646 18632 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dcd0be0c3a94b5197ec871677a5ad9b" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 46531 } }
I20250109 21:03:39.268918 18632 leader_election.cc:304] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dcd0be0c3a94b5197ec871677a5ad9b; no voters:
I20250109 21:03:39.269147 18632 leader_election.cc:290] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:03:39.269253 18637 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:39.269484 18637 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b [term 1 LEADER]: Becoming Leader. State: Replica: 8dcd0be0c3a94b5197ec871677a5ad9b, State: Running, Role: LEADER
I20250109 21:03:39.269780 18637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dcd0be0c3a94b5197ec871677a5ad9b" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 46531 } }
I20250109 21:03:39.270116 18632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:39.271198 18638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dcd0be0c3a94b5197ec871677a5ad9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dcd0be0c3a94b5197ec871677a5ad9b" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 46531 } } }
I20250109 21:03:39.271338 18638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:39.271656 18639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dcd0be0c3a94b5197ec871677a5ad9b. Latest consensus state: current_term: 1 leader_uuid: "8dcd0be0c3a94b5197ec871677a5ad9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dcd0be0c3a94b5197ec871677a5ad9b" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 46531 } } }
I20250109 21:03:39.271749 18639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:39.271886 18646 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:03:39.274691 18646 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:03:39.279963 18646 catalog_manager.cc:1348] Generated new cluster ID: faf569e2b4f840248ae8ffaddaf0d271
I20250109 21:03:39.280019 18646 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:03:39.306133 18646 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:03:39.307132 18646 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:03:39.312090 18646 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 8dcd0be0c3a94b5197ec871677a5ad9b: Generated new TSK 0
I20250109 21:03:39.312707 18646 catalog_manager.cc:1515] Initializing in-progress tserver states...
E20250109 21:03:39.342227 18662 webserver.cc:531] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20250109 21:03:39.342336 18662 webserver.cc:531] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20250109 21:03:39.342936 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:39.325032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38796 (local address 127.12.126.126:46531)
0109 21:03:39.325154 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:39.325158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:39.325264 (+ 106us) server_negotiation.cc:407] Connection header received
0109 21:03:39.325417 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:39.325419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:39.325465 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:39.325563 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0109 21:03:39.325975 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.326488 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:39.327368 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.327500 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:39.327579 (+ 79us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0109 21:03:39.342572 (+ 14993us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41675/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0109 21:03:39.342787 (+ 215us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38796: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41675/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20250109 21:03:39.343011 18635 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38796: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41675/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20250109 21:03:39.343118 18661 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:39.324869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:46531 (local address 127.0.0.1:38796)
0109 21:03:39.325174 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:39.325189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:39.325299 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:39.325579 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:39.325582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:39.325591 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0109 21:03:39.325883 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:39.325891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.326590 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:39.326593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:39.327265 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:39.327276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.327392 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:39.327417 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0109 21:03:39.342765 (+ 15348us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41675/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0109 21:03:39.342951 (+ 186us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.126.126:46531: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41675/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":125,"threads_started":1}
I20250109 21:03:39.343698 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 18565
2025-01-09T21:03:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (291 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20250109 21:03:39.383044 12793 mini_oidc.cc:150] Starting JWKS server
I20250109 21:03:39.383953 12793 webserver.cc:458] Webserver started at https://127.0.0.1:44473/ using document root <none> and password file <none>
I20250109 21:03:39.384040 12793 mini_oidc.cc:186] Starting OIDC Discovery server
I20250109 21:03:39.384277 12793 webserver.cc:458] Webserver started at http://127.0.0.1:37145/ using document root <none> and password file <none>
2025-01-09T21:03:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:03:39Z Disabled control of system clock
I20250109 21:03:39.391714 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:36023
--webserver_interface=127.12.126.126
--webserver_port=0
--builtin_ntp_servers=127.12.126.84:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:36023
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44473/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20250109 21:03:39.493767 18668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:39.493983 18668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:39.494030 18668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:39.496996 18668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:39.497049 18668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:39.497067 18668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:39.497087 18668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:39.497105 18668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250109 21:03:39.497125 18668 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250109 21:03:39.497145 18668 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44473/jwks.json
I20250109 21:03:39.500615 18668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:36023
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:36023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44473/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:39.501605 18668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:39.502651 18668 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:39.509795 18676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:39.510628 18674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:39.511456 18673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:39.512295 18668 server_base.cc:1034] running on GCE node
I20250109 21:03:39.512740 18668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:39.513597 18668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:39.514746 18668 hybrid_clock.cc:648] HybridClock initialized: now 1736456619514735 us; error 25 us; skew 500 ppm
I20250109 21:03:39.516649 18668 webserver.cc:458] Webserver started at http://127.12.126.126:33583/ using document root <none> and password file <none>
I20250109 21:03:39.517189 18668 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:39.517242 18668 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:39.517424 18668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:39.519142 18668 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "8f102c37097a4285ba7769543f5c7674"
format_stamp: "Formatted at 2025-01-09 21:03:39 on dist-test-slave-g0zb"
server_key: "44cf47cd7b3053b1d5c2ff9b461b6548"
server_key_iv: "1d1d3771167163341cc46f027d4c9360"
server_key_version: "encryptionkey@0"
I20250109 21:03:39.519582 18668 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "8f102c37097a4285ba7769543f5c7674"
format_stamp: "Formatted at 2025-01-09 21:03:39 on dist-test-slave-g0zb"
server_key: "44cf47cd7b3053b1d5c2ff9b461b6548"
server_key_iv: "1d1d3771167163341cc46f027d4c9360"
server_key_version: "encryptionkey@0"
I20250109 21:03:39.522615 18668 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:03:39.524626 18682 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:39.525583 18668 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:39.525712 18668 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "8f102c37097a4285ba7769543f5c7674"
format_stamp: "Formatted at 2025-01-09 21:03:39 on dist-test-slave-g0zb"
server_key: "44cf47cd7b3053b1d5c2ff9b461b6548"
server_key_iv: "1d1d3771167163341cc46f027d4c9360"
server_key_version: "encryptionkey@0"
I20250109 21:03:39.525794 18668 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:39.535622 18668 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:39.536223 18668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:39.536399 18668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:39.542922 18734 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:36023 every 8 connection(s)
I20250109 21:03:39.542919 18668 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:36023
I20250109 21:03:39.543890 18668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:03:39.545754 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 18668
I20250109 21:03:39.545862 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:03:39.546114 12793 external_mini_cluster.cc:1453] Setting key 6ee56de7511a799bffe8d5b16c314f62
I20250109 21:03:39.546556 18735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:39.552482 18735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674: Bootstrap starting.
I20250109 21:03:39.553826 18738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:39.547342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46236 (local address 127.12.126.126:36023)
0109 21:03:39.548314 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:39.548321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:39.548354 (+ 33us) server_negotiation.cc:407] Connection header received
0109 21:03:39.548874 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:39.548890 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:39.549198 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:39.549483 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:39.549992 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.550695 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:39.551557 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.551738 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:39.552335 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:39.552387 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:39.552397 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:03:39.552771 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:39.552810 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:03:39.552825 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:39.552904 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:39.553044 (+ 140us) server_negotiation.cc:299] Negotiation successful
0109 21:03:39.553266 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":527,"threads_started":1}
I20250109 21:03:39.554553 18735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:39.555244 18735 log.cc:826] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:39.556288 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:03:39.557147 18735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674: No bootstrap required, opened a new log
I20250109 21:03:39.560194 18735 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f102c37097a4285ba7769543f5c7674" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 36023 } }
I20250109 21:03:39.560427 18735 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:39.560495 18735 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f102c37097a4285ba7769543f5c7674, State: Initialized, Role: FOLLOWER
I20250109 21:03:39.560912 18735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f102c37097a4285ba7769543f5c7674" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 36023 } }
I20250109 21:03:39.561022 18735 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:03:39.561090 18735 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:03:39.561201 18735 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:39.561944 18735 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f102c37097a4285ba7769543f5c7674" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 36023 } }
I20250109 21:03:39.562271 18735 leader_election.cc:304] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f102c37097a4285ba7769543f5c7674; no voters:
I20250109 21:03:39.562525 18735 leader_election.cc:290] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:03:39.562634 18740 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:39.562842 18740 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674 [term 1 LEADER]: Becoming Leader. State: Replica: 8f102c37097a4285ba7769543f5c7674, State: Running, Role: LEADER
I20250109 21:03:39.563169 18740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f102c37097a4285ba7769543f5c7674" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 36023 } }
I20250109 21:03:39.563613 18735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:39.564702 18742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f102c37097a4285ba7769543f5c7674. Latest consensus state: current_term: 1 leader_uuid: "8f102c37097a4285ba7769543f5c7674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f102c37097a4285ba7769543f5c7674" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 36023 } } }
I20250109 21:03:39.564832 18742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:39.564996 18741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f102c37097a4285ba7769543f5c7674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f102c37097a4285ba7769543f5c7674" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 36023 } } }
I20250109 21:03:39.565131 18741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:39.565173 18749 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:03:39.568089 18749 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:03:39.573086 18749 catalog_manager.cc:1348] Generated new cluster ID: 12c47e4ba9cc4eeca12838ed14f9a1fa
I20250109 21:03:39.573150 18749 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:03:39.590749 18749 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:03:39.591653 18749 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:03:39.598966 18749 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 8f102c37097a4285ba7769543f5c7674: Generated new TSK 0
I20250109 21:03:39.599762 18749 catalog_manager.cc:1515] Initializing in-progress tserver states...
E20250109 21:03:39.639551 18765 webserver.cc:531] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20250109 21:03:39.639647 18765 webserver.cc:531] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20250109 21:03:39.640780 18738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:39.617269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.12.126.126:36023)
0109 21:03:39.617398 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:39.617414 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:39.617523 (+ 109us) server_negotiation.cc:407] Connection header received
0109 21:03:39.617693 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:39.617695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:39.617743 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:39.617838 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0109 21:03:39.618247 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.618776 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:39.619665 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.619795 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:39.619877 (+ 82us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0109 21:03:39.640382 (+ 20505us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44473/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0109 21:03:39.640673 (+ 291us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44473/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":39}
W20250109 21:03:39.640837 18738 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44473/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20250109 21:03:39.640920 18764 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:39.617131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:36023 (local address 127.0.0.1:46250)
0109 21:03:39.617434 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:39.617448 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:39.617560 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:39.617851 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:39.617854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:39.617861 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0109 21:03:39.618143 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:39.618150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.618920 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:39.618923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:39.619565 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:39.619576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.619690 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:39.619712 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0109 21:03:39.640587 (+ 20875us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44473/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0109 21:03:39.640745 (+ 158us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.126.126:36023: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44473/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":110,"threads_started":1}
I20250109 21:03:39.641312 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 18668
2025-01-09T21:03:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (297 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2025-01-09T21:03:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:03:39Z Disabled control of system clock
I20250109 21:03:39.688270 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:39773
--webserver_interface=127.12.126.126
--webserver_port=0
--builtin_ntp_servers=127.12.126.84:44509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:39773
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/testchainca.cert with env {}
W20250109 21:03:39.790109 18769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:39.790311 18769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:39.790369 18769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:39.793450 18769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:39.793506 18769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:39.793542 18769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:39.793594 18769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:39.793650 18769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250109 21:03:39.793682 18769 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250109 21:03:39.793712 18769 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20250109 21:03:39.797894 18769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:44509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:39773
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:39773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:39.798882 18769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:39.799700 18769 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:39.807332 18777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:39.808159 18774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:39.808805 18775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:39.809763 18769 server_base.cc:1034] running on GCE node
I20250109 21:03:39.810236 18769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:39.811136 18769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:39.812286 18769 hybrid_clock.cc:648] HybridClock initialized: now 1736456619812279 us; error 26 us; skew 500 ppm
I20250109 21:03:39.814071 18769 webserver.cc:458] Webserver started at http://127.12.126.126:34171/ using document root <none> and password file <none>
I20250109 21:03:39.814605 18769 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:39.814662 18769 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:39.814859 18769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:39.816572 18769 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "a6de681df8af4fab91cbac80aead3f96"
format_stamp: "Formatted at 2025-01-09 21:03:39 on dist-test-slave-g0zb"
server_key: "eaa54be409968409eaccbd043f535566"
server_key_iv: "f589eddf5302734b75dc2cf0c8259469"
server_key_version: "encryptionkey@0"
I20250109 21:03:39.817025 18769 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "a6de681df8af4fab91cbac80aead3f96"
format_stamp: "Formatted at 2025-01-09 21:03:39 on dist-test-slave-g0zb"
server_key: "eaa54be409968409eaccbd043f535566"
server_key_iv: "f589eddf5302734b75dc2cf0c8259469"
server_key_version: "encryptionkey@0"
I20250109 21:03:39.820253 18769 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20250109 21:03:39.822245 18783 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:39.823292 18769 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:39.823410 18769 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "a6de681df8af4fab91cbac80aead3f96"
format_stamp: "Formatted at 2025-01-09 21:03:39 on dist-test-slave-g0zb"
server_key: "eaa54be409968409eaccbd043f535566"
server_key_iv: "f589eddf5302734b75dc2cf0c8259469"
server_key_version: "encryptionkey@0"
I20250109 21:03:39.823522 18769 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:39.835676 18769 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:39.836234 18769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:39.836367 18769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:39.842375 18769 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:39773
I20250109 21:03:39.842382 18835 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:39773 every 8 connection(s)
I20250109 21:03:39.843334 18769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:03:39.846076 18836 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:39.851146 18836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96: Bootstrap starting.
I20250109 21:03:39.853106 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 18769
I20250109 21:03:39.853229 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:03:39.853494 12793 external_mini_cluster.cc:1453] Setting key c08f61ce23bcae23c0e6972e15797f4c
I20250109 21:03:39.853590 18836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:39.854389 18836 log.cc:826] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:39.856217 18836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96: No bootstrap required, opened a new log
I20250109 21:03:39.859210 18836 raft_consensus.cc:357] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6de681df8af4fab91cbac80aead3f96" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 39773 } }
I20250109 21:03:39.859400 18836 raft_consensus.cc:383] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:39.859475 18836 raft_consensus.cc:738] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6de681df8af4fab91cbac80aead3f96, State: Initialized, Role: FOLLOWER
I20250109 21:03:39.859913 18836 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6de681df8af4fab91cbac80aead3f96" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 39773 } }
I20250109 21:03:39.860036 18836 raft_consensus.cc:397] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:03:39.860136 18836 raft_consensus.cc:491] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:03:39.860249 18836 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:39.861171 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:39.854549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.12.126.126:39773)
0109 21:03:39.855286 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:39.855295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:39.855329 (+ 34us) server_negotiation.cc:407] Connection header received
0109 21:03:39.855913 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:39.855929 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:39.856202 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:39.856586 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:39.857047 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.857832 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:39.858570 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.858905 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:39.859390 (+ 485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:39.859412 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:39.859438 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:03:39.859853 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:39.859891 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:03:39.859907 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:39.860002 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:39.860179 (+ 177us) server_negotiation.cc:299] Negotiation successful
0109 21:03:39.860420 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"spinlock_wait_cycles":2432,"thread_start_us":126,"threads_started":1}
I20250109 21:03:39.861223 18836 raft_consensus.cc:513] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6de681df8af4fab91cbac80aead3f96" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 39773 } }
I20250109 21:03:39.861624 18836 leader_election.cc:304] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6de681df8af4fab91cbac80aead3f96; no voters:
I20250109 21:03:39.861894 18836 leader_election.cc:290] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:03:39.861995 18841 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:39.862205 18841 raft_consensus.cc:695] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96 [term 1 LEADER]: Becoming Leader. State: Replica: a6de681df8af4fab91cbac80aead3f96, State: Running, Role: LEADER
I20250109 21:03:39.862617 18841 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6de681df8af4fab91cbac80aead3f96" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 39773 } }
I20250109 21:03:39.863122 18836 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:39.864114 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:03:39.864262 18842 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6de681df8af4fab91cbac80aead3f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6de681df8af4fab91cbac80aead3f96" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 39773 } } }
I20250109 21:03:39.864363 18842 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:39.864763 18849 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:03:39.864295 18841 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6de681df8af4fab91cbac80aead3f96. Latest consensus state: current_term: 1 leader_uuid: "a6de681df8af4fab91cbac80aead3f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6de681df8af4fab91cbac80aead3f96" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 39773 } } }
I20250109 21:03:39.865370 18841 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:39.867877 18849 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:03:39.873553 18849 catalog_manager.cc:1348] Generated new cluster ID: d4117759e8ad4878aed048ce86081196
I20250109 21:03:39.873643 18849 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:03:39.882831 18849 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:03:39.883713 18849 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:03:39.892529 18849 catalog_manager.cc:5899] T 00000000000000000000000000000000 P a6de681df8af4fab91cbac80aead3f96: Generated new TSK 0
I20250109 21:03:39.893327 18849 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:03:39.941342 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:39.903981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55626 (local address 127.12.126.126:39773)
0109 21:03:39.904120 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:39.904123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:39.904244 (+ 121us) server_negotiation.cc:407] Connection header received
0109 21:03:39.904389 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:39.904392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:39.904438 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:39.904536 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0109 21:03:39.904975 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.905474 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:39.906347 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.906492 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:39.906572 (+ 80us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0109 21:03:39.940890 (+ 34318us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0109 21:03:39.941198 (+ 308us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55626: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":65}
W20250109 21:03:39.941416 18840 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55626: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20250109 21:03:39.941496 18865 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:39.903838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:39773 (local address 127.0.0.1:55626)
0109 21:03:39.904144 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:39.904158 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:39.904266 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:39.904550 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:39.904554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:39.904561 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0109 21:03:39.904877 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:39.904885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.905581 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:39.905585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:39.906212 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:39.906222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:39.906360 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:39.906384 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0109 21:03:39.941124 (+ 34740us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0109 21:03:39.941319 (+ 195us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.126.126:39773: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":127,"threads_started":1}
I20250109 21:03:39.942132 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 18769
2025-01-09T21:03:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (280 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (116 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (114 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:03:40 dist-test-slave-g0zb krb5kdc[18877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:03:40 dist-test-slave-g0zb krb5kdc[18877](info): set up 2 sockets
Jan 09 21:03:40 dist-test-slave-g0zb krb5kdc[18877](info): commencing operation
krb5kdc: starting...
W20250109 21:03:41.187798 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:03:41 dist-test-slave-g0zb krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:03:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:03:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:03:41.297869 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:40405
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:40405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:41.400835 18890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:41.401058 18890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:41.401108 18890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:41.404107 18890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:41.404160 18890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:41.404181 18890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:41.404206 18890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:41.404222 18890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:03:41.408316 18890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:40405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:40405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:41.409315 18890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:41.410085 18890 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:41.417871 18895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:41.418725 18896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:41.419471 18898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:41.420454 18890 server_base.cc:1034] running on GCE node
I20250109 21:03:41.420907 18890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:41.421756 18890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:41.422859 18890 hybrid_clock.cc:648] HybridClock initialized: now 1736456621422839 us; error 32 us; skew 500 ppm
Jan 09 21:03:41 dist-test-slave-g0zb krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456621, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:41.425338 18890 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:03:41.426226 18890 webserver.cc:458] Webserver started at http://127.12.126.126:45121/ using document root <none> and password file <none>
I20250109 21:03:41.426889 18890 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:41.426952 18890 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:41.427129 18890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:41.428869 18890 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "4896e1b30e1f4bc390cb99177ec67c49"
format_stamp: "Formatted at 2025-01-09 21:03:41 on dist-test-slave-g0zb"
server_key: "eadcdd24486c1d938228379f562d71b3"
server_key_iv: "da085bbb6d5f6f8529a391366402b6d0"
server_key_version: "encryptionkey@0"
I20250109 21:03:41.429328 18890 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "4896e1b30e1f4bc390cb99177ec67c49"
format_stamp: "Formatted at 2025-01-09 21:03:41 on dist-test-slave-g0zb"
server_key: "eadcdd24486c1d938228379f562d71b3"
server_key_iv: "da085bbb6d5f6f8529a391366402b6d0"
server_key_version: "encryptionkey@0"
I20250109 21:03:41.432523 18890 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:03:41.434588 18905 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:41.435600 18890 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:41.435715 18890 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "4896e1b30e1f4bc390cb99177ec67c49"
format_stamp: "Formatted at 2025-01-09 21:03:41 on dist-test-slave-g0zb"
server_key: "eadcdd24486c1d938228379f562d71b3"
server_key_iv: "da085bbb6d5f6f8529a391366402b6d0"
server_key_version: "encryptionkey@0"
I20250109 21:03:41.435797 18890 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:41.449997 18890 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:41.452517 18890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:41.452674 18890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:41.459112 18890 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:40405
I20250109 21:03:41.459350 18957 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:40405 every 8 connection(s)
I20250109 21:03:41.460043 18890 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:03:41.462373 18958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:41.462779 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 18890
I20250109 21:03:41.462932 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:03:41.463202 12793 external_mini_cluster.cc:1453] Setting key c0f6f70e624637b9a8021db57c075b99
I20250109 21:03:41.467670 18958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49: Bootstrap starting.
I20250109 21:03:41.469871 18958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49: Neither blocks nor log segments found. Creating new log.
Jan 09 21:03:41 dist-test-slave-g0zb krb5kdc[18877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:41.470669 18958 log.cc:826] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:41.472493 18958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49: No bootstrap required, opened a new log
I20250109 21:03:41.474776 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:41.464407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.12.126.126:40405)
0109 21:03:41.464807 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:41.464815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:41.464847 (+ 32us) server_negotiation.cc:407] Connection header received
0109 21:03:41.465390 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:41.465406 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:41.465660 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:41.465958 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:41.466876 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:41.467632 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:41.468304 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:41.468572 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:41.470829 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:41.470850 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:41.470861 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:41.470904 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:41.472599 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:41.472968 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:41.472976 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:41.472980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:41.473046 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:41.473258 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:41.473261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:41.473263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:41.473572 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:41.473740 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:41.473999 (+ 259us) server_negotiation.cc:299] Negotiation successful
0109 21:03:41.474241 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":120,"threads_started":1}
I20250109 21:03:41.475317 18958 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4896e1b30e1f4bc390cb99177ec67c49" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40405 } }
I20250109 21:03:41.475521 18958 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:41.475585 18958 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4896e1b30e1f4bc390cb99177ec67c49, State: Initialized, Role: FOLLOWER
I20250109 21:03:41.475996 18958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4896e1b30e1f4bc390cb99177ec67c49" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40405 } }
I20250109 21:03:41.476111 18958 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:03:41.476169 18958 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:03:41.476284 18958 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:41.477183 18958 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4896e1b30e1f4bc390cb99177ec67c49" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40405 } }
I20250109 21:03:41.477496 18958 leader_election.cc:304] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4896e1b30e1f4bc390cb99177ec67c49; no voters:
I20250109 21:03:41.477752 18958 leader_election.cc:290] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:03:41.477862 18963 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:41.478037 18963 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49 [term 1 LEADER]: Becoming Leader. State: Replica: 4896e1b30e1f4bc390cb99177ec67c49, State: Running, Role: LEADER
I20250109 21:03:41.478370 18963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4896e1b30e1f4bc390cb99177ec67c49" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40405 } }
I20250109 21:03:41.478852 18958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:41.480038 18965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4896e1b30e1f4bc390cb99177ec67c49. Latest consensus state: current_term: 1 leader_uuid: "4896e1b30e1f4bc390cb99177ec67c49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4896e1b30e1f4bc390cb99177ec67c49" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40405 } } }
I20250109 21:03:41.480052 18964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4896e1b30e1f4bc390cb99177ec67c49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4896e1b30e1f4bc390cb99177ec67c49" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40405 } } }
I20250109 21:03:41.480221 18965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:41.480240 18964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:41.480619 18972 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:03:41.483593 18972 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:03:41.488509 18972 catalog_manager.cc:1348] Generated new cluster ID: 89dbafc8e1e446d48632f30a4d31cb8d
I20250109 21:03:41.488564 18972 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:03:41.511479 18972 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:03:41.512710 18972 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:03:41.525602 18972 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 4896e1b30e1f4bc390cb99177ec67c49: Generated new TSK 0
I20250109 21:03:41.526450 18972 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
I20250109 21:03:41.557762 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:40405
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:41.661042 18984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:41.661255 18984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:41.661322 18984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:41.664428 18984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:41.664486 18984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:41.664597 18984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:03:41.668603 18984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:40405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:41.669601 18984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:41.670384 18984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:41.678592 18992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:41.679533 18990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:41.680114 18989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:41.681097 18984 server_base.cc:1034] running on GCE node
I20250109 21:03:41.681442 18984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:41.681964 18984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:41.683105 18984 hybrid_clock.cc:648] HybridClock initialized: now 1736456621683081 us; error 37 us; skew 500 ppm
Jan 09 21:03:41 dist-test-slave-g0zb krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456621, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:41.685707 18984 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:03:41.686601 18984 webserver.cc:458] Webserver started at http://127.12.126.65:42433/ using document root <none> and password file <none>
I20250109 21:03:41.687170 18984 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:41.687232 18984 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:41.687407 18984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:41.689114 18984 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "baf2bbed2293472bb660e4da09bedfcb"
format_stamp: "Formatted at 2025-01-09 21:03:41 on dist-test-slave-g0zb"
server_key: "42300726e02077ce20fcba37b407ff08"
server_key_iv: "40a5d9812437630a2fc64fdd0bca1fc6"
server_key_version: "encryptionkey@0"
I20250109 21:03:41.689572 18984 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "baf2bbed2293472bb660e4da09bedfcb"
format_stamp: "Formatted at 2025-01-09 21:03:41 on dist-test-slave-g0zb"
server_key: "42300726e02077ce20fcba37b407ff08"
server_key_iv: "40a5d9812437630a2fc64fdd0bca1fc6"
server_key_version: "encryptionkey@0"
I20250109 21:03:41.692800 18984 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:03:41.694888 18999 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:41.695859 18984 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:41.695964 18984 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "baf2bbed2293472bb660e4da09bedfcb"
format_stamp: "Formatted at 2025-01-09 21:03:41 on dist-test-slave-g0zb"
server_key: "42300726e02077ce20fcba37b407ff08"
server_key_iv: "40a5d9812437630a2fc64fdd0bca1fc6"
server_key_version: "encryptionkey@0"
I20250109 21:03:41.696048 18984 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:41.709251 18984 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:41.711699 18984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:41.711858 18984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:41.712333 18984 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:41.713136 18984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:41.713186 18984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:41.713244 18984 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:41.713276 18984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:41.721712 18984 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:43921
I20250109 21:03:41.721732 19112 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:43921 every 8 connection(s)
I20250109 21:03:41.722640 18984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:03:41.722968 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 18984
I20250109 21:03:41.723078 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:03:41.723372 12793 external_mini_cluster.cc:1453] Setting key 681a2d0cca0a5de40ad6901d9e2dd522
Jan 09 21:03:41 dist-test-slave-g0zb krb5kdc[18877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456621, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.126.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.66@KRBTEST.COM" created.
I20250109 21:03:41.734709 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:41.724306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:47243 (local address 127.12.126.126:40405)
0109 21:03:41.724431 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:41.724435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:41.725270 (+ 835us) server_negotiation.cc:407] Connection header received
0109 21:03:41.726432 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:41.726438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:41.726539 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:41.726629 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:41.727860 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:41.728330 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:41.728900 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:41.729066 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:41.731473 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:41.731496 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:41.731499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:41.731534 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:41.732784 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:41.733245 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:41.733248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:41.733250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:41.733307 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:41.733566 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:41.733568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:41.733569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:41.733699 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:41.733770 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:41.734336 (+ 566us) server_negotiation.cc:299] Negotiation successful
0109 21:03:41.734470 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250109 21:03:41.735265 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:41.724691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:40405 (local address 127.12.126.65:47243)
0109 21:03:41.725113 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:41.725153 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:41.726235 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:41.726765 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:41.726772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:41.727162 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:41.727704 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:41.727728 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:41.728437 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:41.728440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:41.728800 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:41.728807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:41.729006 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:41.729598 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:41.729622 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:41.731321 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:41.732897 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:41.732901 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:41.732910 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:41.733150 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:41.733403 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:41.733406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:41.733408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:41.733496 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:41.733775 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:41.733782 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:41.734076 (+ 294us) client_negotiation.cc:769] Sending connection context
0109 21:03:41.734292 (+ 216us) client_negotiation.cc:240] Negotiation successful
0109 21:03:41.734521 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":114,"threads_started":1}
I20250109 21:03:41.736507 19113 heartbeater.cc:346] Connected to a master server at 127.12.126.126:40405
I20250109 21:03:41.736727 19113 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:41.737187 19113 heartbeater.cc:510] Master 127.12.126.126:40405 requested a full tablet report, sending...
I20250109 21:03:41.738571 18922 ts_manager.cc:194] Registered new tserver with Master: baf2bbed2293472bb660e4da09bedfcb (127.12.126.65:43921)
I20250109 21:03:41.739719 18922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:47243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
I20250109 21:03:41.747834 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:40405
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:41.849965 19118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:41.850176 19118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:41.850242 19118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:41.853380 19118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:41.853439 19118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:41.853526 19118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:03:41.857651 19118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:40405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:41.858711 19118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:41.859570 19118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:41.867816 19124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:41.868460 19123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:41.869334 19126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:41.870119 19118 server_base.cc:1034] running on GCE node
I20250109 21:03:41.870421 19118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:41.870929 19118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:41.872066 19118 hybrid_clock.cc:648] HybridClock initialized: now 1736456621872051 us; error 29 us; skew 500 ppm
Jan 09 21:03:41 dist-test-slave-g0zb krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456621, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:41.874622 19118 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:03:41.875571 19118 webserver.cc:458] Webserver started at http://127.12.126.66:35173/ using document root <none> and password file <none>
I20250109 21:03:41.876111 19118 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:41.876165 19118 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:41.876349 19118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:41.878152 19118 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "fb3b8d723daa460ea4e493e0a1f7fd4a"
format_stamp: "Formatted at 2025-01-09 21:03:41 on dist-test-slave-g0zb"
server_key: "10812d52cb4ce935ca74c2ecc7962387"
server_key_iv: "352d217bf357669605db5492a2dd544b"
server_key_version: "encryptionkey@0"
I20250109 21:03:41.878609 19118 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "fb3b8d723daa460ea4e493e0a1f7fd4a"
format_stamp: "Formatted at 2025-01-09 21:03:41 on dist-test-slave-g0zb"
server_key: "10812d52cb4ce935ca74c2ecc7962387"
server_key_iv: "352d217bf357669605db5492a2dd544b"
server_key_version: "encryptionkey@0"
I20250109 21:03:41.882082 19118 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20250109 21:03:41.884156 19133 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:41.885121 19118 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:03:41.885218 19118 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "fb3b8d723daa460ea4e493e0a1f7fd4a"
format_stamp: "Formatted at 2025-01-09 21:03:41 on dist-test-slave-g0zb"
server_key: "10812d52cb4ce935ca74c2ecc7962387"
server_key_iv: "352d217bf357669605db5492a2dd544b"
server_key_version: "encryptionkey@0"
I20250109 21:03:41.885313 19118 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:41.894021 19118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:41.896543 19118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:41.896695 19118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:41.897159 19118 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:41.897895 19118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:41.897940 19118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:41.897982 19118 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:41.898011 19118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:41.906128 19118 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:43903
I20250109 21:03:41.906154 19246 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:43903 every 8 connection(s)
I20250109 21:03:41.907124 19118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:03:41.912883 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 19118
I20250109 21:03:41.912997 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:03:41.913219 12793 external_mini_cluster.cc:1453] Setting key 3aab0778e166c31fe05ee8c6edbc09ad
Jan 09 21:03:41 dist-test-slave-g0zb krb5kdc[18877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456621, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:41.918892 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:41.908716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:39013 (local address 127.12.126.126:40405)
0109 21:03:41.908881 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:41.908886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:41.909480 (+ 594us) server_negotiation.cc:407] Connection header received
0109 21:03:41.910602 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:41.910604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:41.910655 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:41.910729 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:41.911882 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:41.912362 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:41.912934 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:41.913110 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:41.915243 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:41.915257 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:41.915265 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:41.915288 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:41.916842 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:41.917419 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:41.917422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:41.917424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:41.917476 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:41.917790 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:41.917792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:41.917793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:41.917924 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:41.917991 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:41.918628 (+ 637us) server_negotiation.cc:299] Negotiation successful
0109 21:03:41.918727 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20250109 21:03:41.919869 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:41.908973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:40405 (local address 127.12.126.66:39013)
0109 21:03:41.909348 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:41.909381 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:41.910413 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:41.910878 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:41.910885 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:41.911229 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:41.911759 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:41.911771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:41.912445 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:41.912447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:41.912818 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:41.912824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:41.913015 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:41.913626 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:41.913649 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:41.915104 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:41.916951 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:41.916959 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:41.916973 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:41.917303 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:41.917560 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:41.917565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:41.917570 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:41.917688 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:41.918010 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:41.918019 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:41.918358 (+ 339us) client_negotiation.cc:769] Sending connection context
0109 21:03:41.918600 (+ 242us) client_negotiation.cc:240] Negotiation successful
0109 21:03:41.918928 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
I20250109 21:03:41.920943 19247 heartbeater.cc:346] Connected to a master server at 127.12.126.126:40405
I20250109 21:03:41.921149 19247 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:41.921627 19247 heartbeater.cc:510] Master 127.12.126.126:40405 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.126.67@KRBTEST.COM; defaulting to no policy
I20250109 21:03:41.922557 18922 ts_manager.cc:194] Registered new tserver with Master: fb3b8d723daa460ea4e493e0a1f7fd4a (127.12.126.66:43903)
I20250109 21:03:41.923125 18922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:39013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
I20250109 21:03:41.939458 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:40405
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:42.045027 19252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:42.045269 19252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:42.045321 19252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:42.048341 19252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:42.048398 19252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:42.048465 19252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:03:42.052443 19252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:40405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:42.053442 19252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:42.054267 19252 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:42.062599 19258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:42.063254 19260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:42.064247 19257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:42.065047 19252 server_base.cc:1034] running on GCE node
I20250109 21:03:42.065403 19252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:42.065944 19252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:42.067086 19252 hybrid_clock.cc:648] HybridClock initialized: now 1736456622067074 us; error 28 us; skew 500 ppm
Jan 09 21:03:42 dist-test-slave-g0zb krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456622, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:42.075906 19252 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:03:42.076822 19252 webserver.cc:458] Webserver started at http://127.12.126.67:42243/ using document root <none> and password file <none>
I20250109 21:03:42.077373 19252 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:42.077426 19252 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:42.077610 19252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:42.079383 19252 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "60b6b627515442e1afb644e3509cd6ce"
format_stamp: "Formatted at 2025-01-09 21:03:42 on dist-test-slave-g0zb"
server_key: "a9eacf7546d3990d87c4a9743b0cf9a9"
server_key_iv: "d4701a4ac7833c389e4cd6600c09122f"
server_key_version: "encryptionkey@0"
I20250109 21:03:42.079800 19252 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "60b6b627515442e1afb644e3509cd6ce"
format_stamp: "Formatted at 2025-01-09 21:03:42 on dist-test-slave-g0zb"
server_key: "a9eacf7546d3990d87c4a9743b0cf9a9"
server_key_iv: "d4701a4ac7833c389e4cd6600c09122f"
server_key_version: "encryptionkey@0"
I20250109 21:03:42.083065 19252 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:03:42.085084 19267 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:42.086061 19252 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:42.086189 19252 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "60b6b627515442e1afb644e3509cd6ce"
format_stamp: "Formatted at 2025-01-09 21:03:42 on dist-test-slave-g0zb"
server_key: "a9eacf7546d3990d87c4a9743b0cf9a9"
server_key_iv: "d4701a4ac7833c389e4cd6600c09122f"
server_key_version: "encryptionkey@0"
I20250109 21:03:42.086280 19252 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:42.096125 19252 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:42.098726 19252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:42.098928 19252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:42.099444 19252 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:42.100279 19252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:42.100325 19252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:42.100383 19252 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:42.100411 19252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:42.108726 19252 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:45295
I20250109 21:03:42.108753 19380 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:45295 every 8 connection(s)
I20250109 21:03:42.109660 19252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:03:42.114904 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 19252
I20250109 21:03:42.115010 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:03:42.115237 12793 external_mini_cluster.cc:1453] Setting key 83c0e55f6cf9b327adee835e1126d383
Jan 09 21:03:42 dist-test-slave-g0zb krb5kdc[18877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456622, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:42.121163 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:42.111244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:51329 (local address 127.12.126.126:40405)
0109 21:03:42.111430 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:42.111435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:42.112103 (+ 668us) server_negotiation.cc:407] Connection header received
0109 21:03:42.113319 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:42.113322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:42.113372 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:42.113449 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:42.114620 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:42.115180 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:42.115841 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:42.115997 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:42.118354 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:42.118368 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:42.118383 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:42.118406 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:42.119778 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:42.120159 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:42.120162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:42.120163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:42.120197 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:42.120382 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:42.120383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:42.120384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:42.120513 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:42.120578 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:42.120958 (+ 380us) server_negotiation.cc:299] Negotiation successful
0109 21:03:42.121049 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20250109 21:03:42.121819 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:42.111534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:40405 (local address 127.12.126.67:51329)
0109 21:03:42.111968 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:42.112000 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:42.113116 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:42.113590 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:42.113597 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:42.113953 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:42.114504 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:42.114516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:42.115293 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:42.115297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:42.115744 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:42.115753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:42.115951 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:42.116661 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:42.116697 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:42.118212 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:42.119869 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:42.119873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:42.119881 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:42.120086 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:42.120254 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:42.120256 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:42.120257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:42.120325 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:42.120580 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:42.120583 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:42.120783 (+ 200us) client_negotiation.cc:769] Sending connection context
0109 21:03:42.120933 (+ 150us) client_negotiation.cc:240] Negotiation successful
0109 21:03:42.121128 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":95,"threads_started":1}
I20250109 21:03:42.122746 19381 heartbeater.cc:346] Connected to a master server at 127.12.126.126:40405
I20250109 21:03:42.122993 19381 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:42.123451 19381 heartbeater.cc:510] Master 127.12.126.126:40405 requested a full tablet report, sending...
I20250109 21:03:42.124383 18922 ts_manager.cc:194] Registered new tserver with Master: 60b6b627515442e1afb644e3509cd6ce (127.12.126.67:45295)
I20250109 21:03:42.124922 18922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:51329
I20250109 21:03:42.128894 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/test-admin.keytab.
Jan 09 21:03:42 dist-test-slave-g0zb krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:42.142912 12793 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20250109 21:03:42.145552 19388 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:42.144197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56576 (local address 127.12.126.65:43921)
0109 21:03:42.144553 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:42.144558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:42.144580 (+ 22us) server_negotiation.cc:407] Connection header received
0109 21:03:42.144639 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:42.144644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:42.144802 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:42.144940 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:42.145374 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56576: BlockingRecv error: recv got EOF from 127.0.0.1:56576 (error 108)
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":127,"threads_started":1}
W20250109 21:03:42.145715 12793 init.cc:244] Error while opening credential cache '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/krb5cc)
Jan 09 21:03:42 dist-test-slave-g0zb krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:42.146657 12793 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jan 09 21:03:42 dist-test-slave-g0zb krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:42.147518 12793 init.cc:295] Successfully reacquired a new kerberos TGT
Jan 09 21:03:42 dist-test-slave-g0zb krb5kdc[18877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.65@KRBTEST.COM
I20250109 21:03:42.155539 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:42.148296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56592 (local address 127.12.126.65:43921)
0109 21:03:42.148404 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:42.148409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:42.148486 (+ 77us) server_negotiation.cc:407] Connection header received
0109 21:03:42.148591 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:42.148594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:42.148633 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:42.148698 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:42.149311 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:42.149984 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:42.150622 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:42.150882 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:42.152614 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:42.152633 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:42.152643 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:42.152664 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:42.154012 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:42.154391 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:42.154396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:42.154397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:42.154445 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:42.154656 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:42.154659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:42.154661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:42.154899 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:42.155099 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:42.155273 (+ 174us) server_negotiation.cc:299] Negotiation successful
0109 21:03:42.155406 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20250109 21:03:42.158738 19388 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:42.157941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56596 (local address 127.12.126.65:43921)
0109 21:03:42.158038 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:42.158041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:42.158161 (+ 120us) server_negotiation.cc:407] Connection header received
0109 21:03:42.158244 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:42.158246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:42.158281 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:42.158343 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:42.158654 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56596: BlockingRecv error: recv got EOF from 127.0.0.1:56596 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20250109 21:03:42.158885 12793 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/krb5cc' of type FILE
Jan 09 21:03:42 dist-test-slave-g0zb krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:42.159739 12793 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jan 09 21:03:42 dist-test-slave-g0zb krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:42.160516 12793 init.cc:295] Successfully reacquired a new kerberos TGT
Jan 09 21:03:42 dist-test-slave-g0zb krb5kdc[18877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.65@KRBTEST.COM
I20250109 21:03:42.167802 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:42.161248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.12.126.65:43921)
0109 21:03:42.161355 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:42.161360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:42.161432 (+ 72us) server_negotiation.cc:407] Connection header received
0109 21:03:42.161560 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:42.161562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:42.161597 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:42.161660 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:42.162222 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:42.162722 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:42.163350 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:42.163580 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:42.165245 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:42.165258 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:42.165260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:42.165283 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:42.166432 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:42.166839 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:42.166843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:42.166844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:42.166884 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:42.167093 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:42.167096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:42.167098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:42.167222 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:42.167335 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:42.167550 (+ 215us) server_negotiation.cc:299] Negotiation successful
0109 21:03:42.167656 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20250109 21:03:42.169978 19388 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:42.169252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.12.126.65:43921)
0109 21:03:42.169366 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:42.169369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:42.169409 (+ 40us) server_negotiation.cc:407] Connection header received
0109 21:03:42.169507 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:42.169509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:42.169540 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:42.169609 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:42.169890 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56624: BlockingRecv error: recv got EOF from 127.0.0.1:56624 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20250109 21:03:42.170195 12793 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1736456573526088-12793-0/krb5kdc/krb5cc' of type FILE
Jan 09 21:03:42 dist-test-slave-g0zb krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:42.171146 12793 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jan 09 21:03:42 dist-test-slave-g0zb krb5kdc[18877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:42.171917 12793 init.cc:295] Successfully reacquired a new kerberos TGT
Jan 09 21:03:42 dist-test-slave-g0zb krb5kdc[18877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.65@KRBTEST.COM
I20250109 21:03:42.179024 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:42.172641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56636 (local address 127.12.126.65:43921)
0109 21:03:42.172744 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:42.172749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:42.172820 (+ 71us) server_negotiation.cc:407] Connection header received
0109 21:03:42.172928 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:42.172931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:42.172966 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:42.173039 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:42.173581 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:42.173999 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:42.174544 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:42.174735 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:42.176522 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:42.176537 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:42.176539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:42.176560 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:42.177739 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:42.178081 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:42.178085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:42.178087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:42.178127 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:42.178320 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:42.178324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:42.178325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:42.178445 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:42.178556 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:42.178757 (+ 201us) server_negotiation.cc:299] Negotiation successful
0109 21:03:42.178873 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20250109 21:03:42.179925 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 18984
I20250109 21:03:42.186225 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 19118
I20250109 21:03:42.191490 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 19252
I20250109 21:03:42.196477 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 18890
2025-01-09T21:03:42Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2025 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:03:42 dist-test-slave-g0zb krb5kdc[19401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:03:42 dist-test-slave-g0zb krb5kdc[19401](info): set up 2 sockets
Jan 09 21:03:42 dist-test-slave-g0zb krb5kdc[19401](info): commencing operation
krb5kdc: starting...
W20250109 21:03:43.219947 12793 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:03:43 dist-test-slave-g0zb krb5kdc[19401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:03:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:03:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.126.keytab.
Entry for principal oryx/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.126.keytab.
I20250109 21:03:43.331801 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:42099
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.126.keytab
--principal=oryx/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:42099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:43.433559 19414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:43.433804 19414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:43.433879 19414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:43.437023 19414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:43.437098 19414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:43.437139 19414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:43.437179 19414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:43.437219 19414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:03:43.441306 19414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:42099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.126.keytab
--principal=oryx/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:42099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:43.442322 19414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:43.443163 19414 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:43.450657 19422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:43.451449 19420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:43.452140 19419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:43.452977 19414 server_base.cc:1034] running on GCE node
I20250109 21:03:43.453444 19414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:43.454305 19414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:43.455457 19414 hybrid_clock.cc:648] HybridClock initialized: now 1736456623455440 us; error 41 us; skew 500 ppm
Jan 09 21:03:43 dist-test-slave-g0zb krb5kdc[19401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456623, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:43.458087 19414 init.cc:377] Logged in from keytab as oryx/127.12.126.126@KRBTEST.COM (short username oryx)
I20250109 21:03:43.458981 19414 webserver.cc:458] Webserver started at http://127.12.126.126:34199/ using document root <none> and password file <none>
I20250109 21:03:43.459502 19414 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:43.459561 19414 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:43.459730 19414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:43.461434 19414 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "77fb9c5a4bed4855be740888c1397e6c"
format_stamp: "Formatted at 2025-01-09 21:03:43 on dist-test-slave-g0zb"
server_key: "64dfef2833927e33884c8ca4e9870822"
server_key_iv: "9d4d54661bbfaf89562d0a5c613637ed"
server_key_version: "encryptionkey@0"
I20250109 21:03:43.461877 19414 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "77fb9c5a4bed4855be740888c1397e6c"
format_stamp: "Formatted at 2025-01-09 21:03:43 on dist-test-slave-g0zb"
server_key: "64dfef2833927e33884c8ca4e9870822"
server_key_iv: "9d4d54661bbfaf89562d0a5c613637ed"
server_key_version: "encryptionkey@0"
I20250109 21:03:43.465248 19414 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20250109 21:03:43.467250 19429 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:43.468181 19414 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:43.468290 19414 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "77fb9c5a4bed4855be740888c1397e6c"
format_stamp: "Formatted at 2025-01-09 21:03:43 on dist-test-slave-g0zb"
server_key: "64dfef2833927e33884c8ca4e9870822"
server_key_iv: "9d4d54661bbfaf89562d0a5c613637ed"
server_key_version: "encryptionkey@0"
I20250109 21:03:43.468369 19414 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:43.479022 19414 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:43.481940 19414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:43.482103 19414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:43.488551 19414 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:42099
I20250109 21:03:43.488786 19481 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:42099 every 8 connection(s)
I20250109 21:03:43.489497 19414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:03:43.491912 19482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:43.496701 19482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c: Bootstrap starting.
I20250109 21:03:43.496737 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 19414
I20250109 21:03:43.496842 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:03:43.497100 12793 external_mini_cluster.cc:1453] Setting key 4ef5c50219b85419a266a68ec3ad2208
I20250109 21:03:43.499104 19482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:43.499780 19482 log.cc:826] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:43.501394 19482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c: No bootstrap required, opened a new log
Jan 09 21:03:43 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM
I20250109 21:03:43.504791 19482 raft_consensus.cc:357] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } }
I20250109 21:03:43.504988 19482 raft_consensus.cc:383] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:43.505031 19482 raft_consensus.cc:738] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77fb9c5a4bed4855be740888c1397e6c, State: Initialized, Role: FOLLOWER
I20250109 21:03:43.505415 19482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } }
I20250109 21:03:43.505527 19482 raft_consensus.cc:397] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:03:43.505586 19482 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:03:43.505694 19482 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:43.506429 19482 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } }
I20250109 21:03:43.506759 19482 leader_election.cc:304] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77fb9c5a4bed4855be740888c1397e6c; no voters:
I20250109 21:03:43.507071 19482 leader_election.cc:290] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:03:43.507138 19487 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:43.507359 19487 raft_consensus.cc:695] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [term 1 LEADER]: Becoming Leader. State: Replica: 77fb9c5a4bed4855be740888c1397e6c, State: Running, Role: LEADER
I20250109 21:03:43.507689 19487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } }
I20250109 21:03:43.508240 19482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:43.509393 19488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } }
I20250109 21:03:43.509527 19488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:43.509780 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:43.498206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.12.126.126:42099)
0109 21:03:43.498629 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:43.498638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:43.498670 (+ 32us) server_negotiation.cc:407] Connection header received
0109 21:03:43.499242 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:43.499268 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:43.499550 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:43.499862 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:43.500713 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:43.501572 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:43.502184 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:43.502443 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:43.504872 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:43.504889 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:43.504899 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:43.504929 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:43.506838 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:43.507342 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:43.507348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:43.507353 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:43.507443 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:43.507711 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:43.507714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:43.507716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:43.508124 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:43.508319 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:43.508783 (+ 464us) server_negotiation.cc:299] Negotiation successful
0109 21:03:43.509029 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":119,"threads_started":1}
I20250109 21:03:43.509863 19496 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:03:43.509786 19489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77fb9c5a4bed4855be740888c1397e6c. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } }
I20250109 21:03:43.509907 19489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:43.512785 19496 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:03:43.517917 19496 catalog_manager.cc:1348] Generated new cluster ID: 4bd8a54b43d446b0b0d221053a3d2adc
I20250109 21:03:43.517975 19496 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:03:43.537890 19496 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:03:43.538848 19496 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:03:43.543936 19496 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c: Generated new TSK 0
I20250109 21:03:43.544564 19496 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.65.keytab.
Entry for principal oryx/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.65.keytab.
I20250109 21:03:43.573230 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42099
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.65.keytab
--principal=oryx/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:43.685391 19508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:43.685672 19508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:43.685729 19508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:43.688884 19508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:43.688949 19508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:43.689060 19508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:03:43.693115 19508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.65.keytab
--principal=oryx/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:43.694175 19508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:43.695046 19508 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:43.703284 19513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:43.703876 19514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:43.704731 19516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:43.705508 19508 server_base.cc:1034] running on GCE node
I20250109 21:03:43.705854 19508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:43.706353 19508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:43.707492 19508 hybrid_clock.cc:648] HybridClock initialized: now 1736456623707448 us; error 60 us; skew 500 ppm
Jan 09 21:03:43 dist-test-slave-g0zb krb5kdc[19401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456623, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:43.710098 19508 init.cc:377] Logged in from keytab as oryx/127.12.126.65@KRBTEST.COM (short username oryx)
I20250109 21:03:43.711082 19508 webserver.cc:458] Webserver started at http://127.12.126.65:46061/ using document root <none> and password file <none>
I20250109 21:03:43.711634 19508 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:43.711690 19508 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:43.711856 19508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:43.713618 19508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "ffe52c1a4d014d29a2a57c3120d158b1"
format_stamp: "Formatted at 2025-01-09 21:03:43 on dist-test-slave-g0zb"
server_key: "bc0641937987920803b6e969705111c2"
server_key_iv: "3f3e942cab674f9e896250bf2a582cd8"
server_key_version: "encryptionkey@0"
I20250109 21:03:43.714063 19508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "ffe52c1a4d014d29a2a57c3120d158b1"
format_stamp: "Formatted at 2025-01-09 21:03:43 on dist-test-slave-g0zb"
server_key: "bc0641937987920803b6e969705111c2"
server_key_iv: "3f3e942cab674f9e896250bf2a582cd8"
server_key_version: "encryptionkey@0"
I20250109 21:03:43.717252 19508 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250109 21:03:43.719385 19523 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:43.720340 19508 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 21:03:43.720440 19508 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "ffe52c1a4d014d29a2a57c3120d158b1"
format_stamp: "Formatted at 2025-01-09 21:03:43 on dist-test-slave-g0zb"
server_key: "bc0641937987920803b6e969705111c2"
server_key_iv: "3f3e942cab674f9e896250bf2a582cd8"
server_key_version: "encryptionkey@0"
I20250109 21:03:43.720531 19508 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:43.737639 19508 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:43.740180 19508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:43.740338 19508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:43.740823 19508 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:43.741583 19508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:43.741626 19508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:43.741680 19508 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:43.741703 19508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:43.750031 19508 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:35925
I20250109 21:03:43.750053 19636 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:35925 every 8 connection(s)
I20250109 21:03:43.750984 19508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
Jan 09 21:03:43 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456623, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM
I20250109 21:03:43.759542 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 19508
I20250109 21:03:43.759665 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:03:43.759927 12793 external_mini_cluster.cc:1453] Setting key 962c6bb953adb822299cc3435a7b3be8
I20250109 21:03:43.762034 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:43.752577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:47555 (local address 127.12.126.126:42099)
0109 21:03:43.752698 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:43.752702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:43.753710 (+ 1008us) server_negotiation.cc:407] Connection header received
0109 21:03:43.754444 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:43.754446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:43.754496 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:43.754578 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:43.755821 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:43.756299 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:43.756857 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:43.756975 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:43.758929 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:43.758947 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:43.758950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:43.758974 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:43.760267 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:43.760742 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:43.760745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:43.760747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:43.760790 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:43.761021 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:43.761024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:43.761025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:43.761199 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:43.761300 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:43.761818 (+ 518us) server_negotiation.cc:299] Negotiation successful
0109 21:03:43.761920 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250109 21:03:43.763003 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:43.753187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.65:47555)
0109 21:03:43.753573 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:43.753604 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:43.754267 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:43.754715 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:43.754722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:43.755121 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:43.755692 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:43.755704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:43.756389 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:43.756391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:43.756770 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:43.756777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:43.756934 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:43.757474 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:43.757495 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:43.758777 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:43.760367 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:43.760371 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:43.760379 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:43.760665 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:43.760869 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:43.760871 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:43.760872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:43.760957 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:43.761310 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:43.761316 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:43.761579 (+ 263us) client_negotiation.cc:769] Sending connection context
0109 21:03:43.761799 (+ 220us) client_negotiation.cc:240] Negotiation successful
0109 21:03:43.762078 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":92,"threads_started":1}
I20250109 21:03:43.764145 19637 heartbeater.cc:346] Connected to a master server at 127.12.126.126:42099
I20250109 21:03:43.764380 19637 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:43.764984 19637 heartbeater.cc:510] Master 127.12.126.126:42099 requested a full tablet report, sending...
I20250109 21:03:43.766439 19446 ts_manager.cc:194] Registered new tserver with Master: ffe52c1a4d014d29a2a57c3120d158b1 (127.12.126.65:35925)
I20250109 21:03:43.767632 19446 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:47555
WARNING: no policy specified for oryx/127.12.126.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.126.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.66.keytab.
Entry for principal oryx/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.66.keytab.
I20250109 21:03:43.783891 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42099
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.66.keytab
--principal=oryx/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:43.885810 19642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:43.886075 19642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:43.886181 19642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:43.889333 19642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:43.889412 19642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:43.889526 19642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:03:43.893496 19642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.66.keytab
--principal=oryx/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:43.894515 19642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:43.895363 19642 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:43.903705 19648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:43.904291 19650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:43.905186 19647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:43.905956 19642 server_base.cc:1034] running on GCE node
I20250109 21:03:43.906345 19642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:43.906924 19642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:43.908079 19642 hybrid_clock.cc:648] HybridClock initialized: now 1736456623908069 us; error 24 us; skew 500 ppm
Jan 09 21:03:43 dist-test-slave-g0zb krb5kdc[19401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456623, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:43.910830 19642 init.cc:377] Logged in from keytab as oryx/127.12.126.66@KRBTEST.COM (short username oryx)
I20250109 21:03:43.911670 19642 webserver.cc:458] Webserver started at http://127.12.126.66:34881/ using document root <none> and password file <none>
I20250109 21:03:43.912199 19642 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:43.912245 19642 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:43.912392 19642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:43.914084 19642 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "44684bbb8586432aa2cc81943640a975"
format_stamp: "Formatted at 2025-01-09 21:03:43 on dist-test-slave-g0zb"
server_key: "c7bf889594351f2eaf95085de263bbb9"
server_key_iv: "96f5a2128bde31e39237ce33d7ff459f"
server_key_version: "encryptionkey@0"
I20250109 21:03:43.914505 19642 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "44684bbb8586432aa2cc81943640a975"
format_stamp: "Formatted at 2025-01-09 21:03:43 on dist-test-slave-g0zb"
server_key: "c7bf889594351f2eaf95085de263bbb9"
server_key_iv: "96f5a2128bde31e39237ce33d7ff459f"
server_key_version: "encryptionkey@0"
I20250109 21:03:43.917622 19642 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:03:43.919680 19657 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:43.920589 19642 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:43.920675 19642 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "44684bbb8586432aa2cc81943640a975"
format_stamp: "Formatted at 2025-01-09 21:03:43 on dist-test-slave-g0zb"
server_key: "c7bf889594351f2eaf95085de263bbb9"
server_key_iv: "96f5a2128bde31e39237ce33d7ff459f"
server_key_version: "encryptionkey@0"
I20250109 21:03:43.920763 19642 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:43.941190 19642 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:43.943910 19642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:43.944078 19642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:43.944612 19642 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:43.945382 19642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:43.945432 19642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:43.945488 19642 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:43.945521 19642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:43.954066 19642 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:45203
I20250109 21:03:43.954098 19770 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:45203 every 8 connection(s)
I20250109 21:03:43.955086 19642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:03:43.959201 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 19642
I20250109 21:03:43.959344 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:03:43.959640 12793 external_mini_cluster.cc:1453] Setting key ed95a2bfbe1f350485bf2277c8499193
Jan 09 21:03:43 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456623, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM
I20250109 21:03:43.968662 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:43.956765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:40897 (local address 127.12.126.126:42099)
0109 21:03:43.956920 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:43.956924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:43.958024 (+ 1100us) server_negotiation.cc:407] Connection header received
0109 21:03:43.958861 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:43.958865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:43.958921 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:43.959002 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:43.960660 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:43.961296 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:43.962205 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:43.962338 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:43.964968 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:43.964987 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:43.964989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:43.965016 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:43.966567 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:43.967244 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:43.967247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:43.967248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:43.967295 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:43.967608 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:43.967610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:43.967611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:43.967751 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:43.967825 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:43.968444 (+ 619us) server_negotiation.cc:299] Negotiation successful
0109 21:03:43.968541 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for oryx/127.12.126.67@KRBTEST.COM; defaulting to no policy
I20250109 21:03:43.969612 19773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:43.957457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.66:40897)
0109 21:03:43.957858 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:43.957892 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:43.958607 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:43.959203 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:43.959214 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:43.959792 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:43.960524 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:43.960540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:43.961425 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:43.961430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:43.962076 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:43.962089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:43.962347 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:43.963080 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:43.963105 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:43.964782 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:43.966691 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:43.966701 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:43.966716 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:43.967135 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:43.967388 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:43.967394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:43.967397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:43.967518 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:43.967846 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:43.967853 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:43.968189 (+ 336us) client_negotiation.cc:769] Sending connection context
0109 21:03:43.968435 (+ 246us) client_negotiation.cc:240] Negotiation successful
0109 21:03:43.968749 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":98,"threads_started":1}
I20250109 21:03:43.970683 19771 heartbeater.cc:346] Connected to a master server at 127.12.126.126:42099
I20250109 21:03:43.970937 19771 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:43.971562 19771 heartbeater.cc:510] Master 127.12.126.126:42099 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.126.67@KRBTEST.COM" created.
I20250109 21:03:43.972730 19446 ts_manager.cc:194] Registered new tserver with Master: 44684bbb8586432aa2cc81943640a975 (127.12.126.66:45203)
I20250109 21:03:43.973491 19446 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:40897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.67.keytab.
Entry for principal oryx/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.67.keytab.
I20250109 21:03:43.985692 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42099
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.67.keytab
--principal=oryx/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:44.089284 19776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:44.089512 19776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:44.089563 19776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:44.092360 19776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:44.092427 19776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:44.092494 19776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:03:44.096285 19776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.67.keytab
--principal=oryx/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:44.097251 19776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:44.098039 19776 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:44.106175 19784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:44.106709 19782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:44.107658 19781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:44.108472 19776 server_base.cc:1034] running on GCE node
I20250109 21:03:44.108832 19776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:44.109395 19776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:44.110535 19776 hybrid_clock.cc:648] HybridClock initialized: now 1736456624110512 us; error 36 us; skew 500 ppm
Jan 09 21:03:44 dist-test-slave-g0zb krb5kdc[19401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456624, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:44.113408 19776 init.cc:377] Logged in from keytab as oryx/127.12.126.67@KRBTEST.COM (short username oryx)
I20250109 21:03:44.114284 19776 webserver.cc:458] Webserver started at http://127.12.126.67:43869/ using document root <none> and password file <none>
I20250109 21:03:44.114859 19776 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:44.114918 19776 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:44.115106 19776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:44.116910 19776 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "972591450de34e58b065db92abc49a77"
format_stamp: "Formatted at 2025-01-09 21:03:44 on dist-test-slave-g0zb"
server_key: "bba422824c9f1512547ae8739b51e64a"
server_key_iv: "57f3d5b726891e4981b5b146a83d07a8"
server_key_version: "encryptionkey@0"
I20250109 21:03:44.117380 19776 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "972591450de34e58b065db92abc49a77"
format_stamp: "Formatted at 2025-01-09 21:03:44 on dist-test-slave-g0zb"
server_key: "bba422824c9f1512547ae8739b51e64a"
server_key_iv: "57f3d5b726891e4981b5b146a83d07a8"
server_key_version: "encryptionkey@0"
I20250109 21:03:44.120663 19776 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:03:44.122854 19791 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:44.123837 19776 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:44.123934 19776 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "972591450de34e58b065db92abc49a77"
format_stamp: "Formatted at 2025-01-09 21:03:44 on dist-test-slave-g0zb"
server_key: "bba422824c9f1512547ae8739b51e64a"
server_key_iv: "57f3d5b726891e4981b5b146a83d07a8"
server_key_version: "encryptionkey@0"
I20250109 21:03:44.124025 19776 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:44.144277 19776 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:44.146899 19776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:44.147064 19776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:44.147598 19776 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:44.148347 19776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:44.148391 19776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:44.148433 19776 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:44.148461 19776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:44.156468 19776 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:36481
I20250109 21:03:44.156494 19904 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:36481 every 8 connection(s)
I20250109 21:03:44.157397 19776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:03:44.161124 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 19776
I20250109 21:03:44.161222 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:03:44.161441 12793 external_mini_cluster.cc:1453] Setting key 918e08a866b53f387e50c259b17bcc60
Jan 09 21:03:44 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456624, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM
I20250109 21:03:44.168638 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.159062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:50105 (local address 127.12.126.126:42099)
0109 21:03:44.159217 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.159220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.160205 (+ 985us) server_negotiation.cc:407] Connection header received
0109 21:03:44.160974 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.160977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.161039 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.161111 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:44.162490 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.163123 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.163766 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.163896 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.165889 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:44.165903 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:44.165905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:44.165929 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:44.167197 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.167617 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.167619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.167621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.167656 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.167849 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.167850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.167852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.167986 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:44.168052 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.168426 (+ 374us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.168526 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250109 21:03:44.169224 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.159701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.67:50105)
0109 21:03:44.160076 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.160108 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:44.160763 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:44.161291 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:44.161303 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:44.161685 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:44.162360 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.162375 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.163242 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.163248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:44.163664 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.163673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.163836 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.164356 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:44.164376 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:44.165758 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:44.167295 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.167299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.167308 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.167541 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.167717 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.167719 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.167720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.167788 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.168050 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:44.168054 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:44.168252 (+ 198us) client_negotiation.cc:769] Sending connection context
0109 21:03:44.168402 (+ 150us) client_negotiation.cc:240] Negotiation successful
0109 21:03:44.168607 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":94,"threads_started":1}
I20250109 21:03:44.170233 19905 heartbeater.cc:346] Connected to a master server at 127.12.126.126:42099
I20250109 21:03:44.170437 19905 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:44.170935 19905 heartbeater.cc:510] Master 127.12.126.126:42099 requested a full tablet report, sending...
I20250109 21:03:44.171862 19446 ts_manager.cc:194] Registered new tserver with Master: 972591450de34e58b065db92abc49a77 (127.12.126.67:36481)
I20250109 21:03:44.172398 19446 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:50105
I20250109 21:03:44.174921 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.126.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.126.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.126.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.125.keytab.
Entry for principal oryx/127.12.126.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.125.keytab.
I20250109 21:03:44.198212 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.125:40035
--webserver_interface=127.12.126.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.125.keytab
--principal=oryx/127.12.126.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:42099,127.12.126.125:40035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:44.300259 19910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:44.300532 19910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:44.300640 19910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:44.303911 19910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:44.303972 19910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:44.304006 19910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:44.304049 19910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:44.304081 19910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:03:44.308202 19910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:42099,127.12.126.125:40035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.125.keytab
--principal=oryx/127.12.126.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.125:40035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.126.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
W20250109 21:03:44.308416 19910 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.126.126:42099,127.12.126.125:40035'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20250109 21:03:44.309260 19910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:44.310112 19910 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:44.317310 19916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:44.318349 19915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:44.319064 19918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:44.319981 19910 server_base.cc:1034] running on GCE node
I20250109 21:03:44.320451 19910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:44.321316 19910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:44.322456 19910 hybrid_clock.cc:648] HybridClock initialized: now 1736456624322454 us; error 34 us; skew 500 ppm
Jan 09 21:03:44 dist-test-slave-g0zb krb5kdc[19401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456624, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:44.325299 19910 init.cc:377] Logged in from keytab as oryx/127.12.126.125@KRBTEST.COM (short username oryx)
I20250109 21:03:44.326112 19910 webserver.cc:458] Webserver started at http://127.12.126.125:41385/ using document root <none> and password file <none>
I20250109 21:03:44.326632 19910 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:44.326681 19910 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:44.326900 19910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:44.328692 19910 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/data/instance:
uuid: "fed07e4fff2e449a95dd07aa8cf7925f"
format_stamp: "Formatted at 2025-01-09 21:03:44 on dist-test-slave-g0zb"
server_key: "ae44ae6209beb83265450c3784e4f598"
server_key_iv: "d3ffad3ebcffa06f5377de90464afa92"
server_key_version: "encryptionkey@0"
I20250109 21:03:44.329098 19910 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/wal/instance:
uuid: "fed07e4fff2e449a95dd07aa8cf7925f"
format_stamp: "Formatted at 2025-01-09 21:03:44 on dist-test-slave-g0zb"
server_key: "ae44ae6209beb83265450c3784e4f598"
server_key_iv: "d3ffad3ebcffa06f5377de90464afa92"
server_key_version: "encryptionkey@0"
I20250109 21:03:44.332183 19910 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:03:44.334131 19925 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:44.335105 19910 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:44.335196 19910 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/wal
uuid: "fed07e4fff2e449a95dd07aa8cf7925f"
format_stamp: "Formatted at 2025-01-09 21:03:44 on dist-test-slave-g0zb"
server_key: "ae44ae6209beb83265450c3784e4f598"
server_key_iv: "d3ffad3ebcffa06f5377de90464afa92"
server_key_version: "encryptionkey@0"
I20250109 21:03:44.335264 19910 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:44.348400 19910 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:44.350777 19910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:44.350950 19910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:44.356889 19910 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.125:40035
I20250109 21:03:44.357115 19977 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.125:40035 every 8 connection(s)
I20250109 21:03:44.357851 19910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/data/info.pb
I20250109 21:03:44.360360 19978 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:44.363653 19978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } has no permanent_uuid. Determining permanent_uuid...
I20250109 21:03:44.364049 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 19910
Jan 09 21:03:44 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456624, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.125@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM
Jan 09 21:03:44 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.126.125@KRBTEST.COM
I20250109 21:03:44.374877 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.365305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51280 (local address 127.12.126.126:42099)
0109 21:03:44.365467 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.365471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.365881 (+ 410us) server_negotiation.cc:407] Connection header received
0109 21:03:44.366586 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.366589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.366639 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.366729 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:44.367970 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.368452 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.369012 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.369188 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.371382 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:44.371400 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:44.371403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:44.371429 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:44.372822 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.373329 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.373333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.373335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.373381 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.373649 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.373652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.373654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.373859 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:44.373962 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.374490 (+ 528us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.374624 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20250109 21:03:44.375525 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.365332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.0.0.1:51280)
0109 21:03:44.365756 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.365792 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:44.366398 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:44.367187 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:44.367191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:44.367546 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:44.367815 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.367824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.368565 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.368568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:44.368908 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.368914 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.369111 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.369722 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:44.369743 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:44.371206 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:44.372926 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.372934 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.372942 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.373242 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.373483 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.373488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.373490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.373568 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.373955 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:44.373962 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:44.374244 (+ 282us) client_negotiation.cc:769] Sending connection context
0109 21:03:44.374417 (+ 173us) client_negotiation.cc:240] Negotiation successful
0109 21:03:44.374672 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"spinlock_wait_cycles":14976,"thread_start_us":112,"threads_started":1}
I20250109 21:03:44.375525 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.365332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53270 (local address 127.12.126.125:40035)
0109 21:03:44.365724 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.365732 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.365763 (+ 31us) server_negotiation.cc:407] Connection header received
0109 21:03:44.366385 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.366403 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.366596 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.366797 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:44.367619 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.368326 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.368922 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.369156 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.371716 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:44.371732 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:44.371740 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:44.371764 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:44.373367 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.373742 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.373745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.373747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.373785 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.373990 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.373993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.373994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.374252 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:44.374432 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.374683 (+ 251us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.374809 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"spinlock_wait_cycles":55296,"thread_start_us":143,"threads_started":1}
I20250109 21:03:44.376802 19978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } has no permanent_uuid. Determining permanent_uuid...
Jan 09 21:03:44 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456624, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.125@KRBTEST.COM for oryx/127.12.126.125@KRBTEST.COM
I20250109 21:03:44.383716 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.377278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:40035 (local address 127.0.0.1:53274)
0109 21:03:44.377480 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.377491 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:44.377560 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:44.377756 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:44.377759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:44.377897 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:44.378061 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.378066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.378690 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.378693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:44.379055 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.379062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.379158 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.379622 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:44.379633 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:44.380731 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:44.382657 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.382660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.382661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.382869 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.383048 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.383050 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.383051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.383089 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.383333 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:44.383334 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:44.383396 (+ 62us) client_negotiation.cc:769] Sending connection context
0109 21:03:44.383488 (+ 92us) client_negotiation.cc:240] Negotiation successful
0109 21:03:44.383585 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20250109 21:03:44.383769 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.377488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53274 (local address 127.12.126.125:40035)
0109 21:03:44.377591 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.377596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.377611 (+ 15us) server_negotiation.cc:407] Connection header received
0109 21:03:44.377639 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.377642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.377675 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.377737 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:44.378158 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.378599 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.379148 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.379338 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.380851 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:44.380864 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:44.380867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:44.380886 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:44.382558 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.382944 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.382946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.382948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.382984 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.383144 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.383146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.383147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.383272 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:44.383338 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.383528 (+ 190us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.383638 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":41}
I20250109 21:03:44.386247 19978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: Bootstrap starting.
I20250109 21:03:44.388379 19978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:44.389010 19978 log.cc:826] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:44.390638 19978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: No bootstrap required, opened a new log
I20250109 21:03:44.393072 19978 raft_consensus.cc:357] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } }
I20250109 21:03:44.393245 19978 raft_consensus.cc:383] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:44.393281 19978 raft_consensus.cc:738] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed07e4fff2e449a95dd07aa8cf7925f, State: Initialized, Role: FOLLOWER
I20250109 21:03:44.393623 19978 consensus_queue.cc:260] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } }
I20250109 21:03:44.393996 19984 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } } }
I20250109 21:03:44.394101 19984 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:03:44.394582 19978 sys_catalog.cc:564] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [sys.catalog]: configured and running, proceeding with master startup.
W20250109 21:03:44.398726 19995 catalog_manager.cc:1559] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:03:44.398818 19995 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 21:03:44.400565 19910 master_runner.cc:428] Detected that this master fed07e4fff2e449a95dd07aa8cf7925f is joining an existing cluster
I20250109 21:03:44.400635 19910 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.126.125:40035
I20250109 21:03:44.409372 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.402914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51288 (local address 127.12.126.126:42099)
0109 21:03:44.403171 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.403175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.403381 (+ 206us) server_negotiation.cc:407] Connection header received
0109 21:03:44.403598 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.403604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.403652 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.403725 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:44.404294 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.405018 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.405545 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.405759 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.406563 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:44.406579 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:44.406581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:44.406607 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:44.408064 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.408477 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.408480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.408481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.408519 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.408731 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.408733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.408734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.408874 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:44.408941 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.409173 (+ 232us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.409261 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20250109 21:03:44.409435 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.402776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.0.0.1:51288)
0109 21:03:44.403300 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.403314 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:44.403445 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:44.403733 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:44.403736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:44.403935 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:44.404171 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.404177 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.405089 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.405095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:44.405444 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.405450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.405567 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.406166 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:44.406179 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:44.406443 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:44.408181 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.408184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.408186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.408390 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.408590 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.408593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.408595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.408661 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.408948 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:44.408952 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:44.409037 (+ 85us) client_negotiation.cc:769] Sending connection context
0109 21:03:44.409144 (+ 107us) client_negotiation.cc:240] Negotiation successful
0109 21:03:44.409269 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":126,"threads_started":1}
Jan 09 21:03:44 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456623, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.126@KRBTEST.COM for oryx/127.12.126.125@KRBTEST.COM
I20250109 21:03:44.431344 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.402856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.12.126.125:40035)
0109 21:03:44.402980 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.402984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.403382 (+ 398us) server_negotiation.cc:407] Connection header received
0109 21:03:44.403500 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.403505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.403539 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.403612 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:44.404303 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.404946 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.405704 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.405870 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.428126 (+ 22256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:44.428142 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:44.428145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:44.428167 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:44.429788 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.430176 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.430181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.430184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.430233 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.430452 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.430454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.430456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.430634 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:44.430705 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.431054 (+ 349us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.431172 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39,"spinlock_wait_cycles":2304}
I20250109 21:03:44.431398 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.413346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:40035 (local address 127.0.0.1:53298)
0109 21:03:44.413666 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.413694 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:44.413786 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:44.414204 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:44.414209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:44.414646 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:44.414927 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.414937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.415623 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.415626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:44.416168 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.416178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.416289 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.416798 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:44.416814 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:44.418340 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:44.429602 (+ 11262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.429617 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.429624 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.429973 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.430240 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.430243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.430245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.430313 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.430758 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:44.430765 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:44.430941 (+ 176us) client_negotiation.cc:769] Sending connection context
0109 21:03:44.431077 (+ 136us) client_negotiation.cc:240] Negotiation successful
0109 21:03:44.431242 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":97,"threads_started":1}
I20250109 21:03:44.431947 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.413385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53298 (local address 127.12.126.125:40035)
0109 21:03:44.413982 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.413984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.413993 (+ 9us) server_negotiation.cc:407] Connection header received
0109 21:03:44.414036 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.414038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.414074 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.414142 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:44.415061 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.415532 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.416264 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.416406 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.418464 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:44.418484 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:44.418486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:44.418507 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:44.429449 (+ 10942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.430092 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.430095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.430097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.430143 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.430420 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.430425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.430428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.430634 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:44.431598 (+ 964us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.431680 (+ 82us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.431779 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"spinlock_wait_cycles":5248,"thread_start_us":90,"threads_started":1}
I20250109 21:03:44.432598 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.403041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:40035 (local address 127.0.0.1:53286)
0109 21:03:44.403299 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.403310 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:44.403407 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:44.403641 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:44.403643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:44.403889 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:44.404215 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.404223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.405041 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.405045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:44.405585 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.405594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.405962 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.427427 (+ 21465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:44.427448 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:44.427949 (+ 501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:44.429893 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.429898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.429900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.430090 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.430318 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.430320 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.430322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.430370 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.430760 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:44.430762 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:44.430867 (+ 105us) client_negotiation.cc:769] Sending connection context
0109 21:03:44.432340 (+ 1473us) client_negotiation.cc:240] Negotiation successful
0109 21:03:44.432459 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
I20250109 21:03:44.433305 19446 catalog_manager.cc:7010] Initiating ChangeConfig request to add master 127.12.126.125:40035: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f"
member_type: NON_VOTER
last_known_addr {
host: "127.12.126.125"
port: 40035
}
attrs {
promote: true
}
}
dest_uuid: "77fb9c5a4bed4855be740888c1397e6c"
cas_config_opid_index: -1
I20250109 21:03:44.433948 19446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } }
I20250109 21:03:44.435672 20007 raft_consensus.cc:2949] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fed07e4fff2e449a95dd07aa8cf7925f (127.12.126.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } } }
I20250109 21:03:44.436275 19952 raft_consensus.cc:3054] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:44.437533 20007 catalog_manager.cc:7014] Successfully completed master ChangeConfig request to add master 127.12.126.125:40035
I20250109 21:03:44.437527 19952 raft_consensus.cc:1270] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 1 FOLLOWER]: Refusing update from remote peer 77fb9c5a4bed4855be740888c1397e6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250109 21:03:44.437927 19910 master.cc:537] Master@127.12.126.125:40035 shutting down...
I20250109 21:03:44.438325 20007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } } }
I20250109 21:03:44.438359 20006 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:44.438501 20007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: This master's current role is: LEADER
W20250109 21:03:44.440415 19433 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20250109 21:03:44.441958 19910 raft_consensus.cc:2238] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 1 FOLLOWER]: Raft consensus shutting down.
WARNING: no policy specified for oryx/127.12.126.124@KRBTEST.COM; defaulting to no policy
I20250109 21:03:44.442207 19910 raft_consensus.cc:2267] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:44.442315 19910 tablet_replica.cc:331] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: stopping tablet replica
W20250109 21:03:44.443133 19433 consensus_peers.cc:487] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c -> Peer fed07e4fff2e449a95dd07aa8cf7925f (127.12.126.125:40035): Couldn't send request to peer fed07e4fff2e449a95dd07aa8cf7925f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.126.124@KRBTEST.COM" created.
I20250109 21:03:44.446671 19910 master.cc:559] Master@127.12.126.125:40035 shutdown complete.
I20250109 21:03:44.446722 19910 master_runner.cc:305] Clearing existing system tablet
I20250109 21:03:44.446986 19910 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:44.447041 19910 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:44.448350 19910 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20250109 21:03:44.449258 20014 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:44.449483 19910 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:44.449628 19910 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250109 21:03:44.449699 19910 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/wal
uuid: "fed07e4fff2e449a95dd07aa8cf7925f"
format_stamp: "Formatted at 2025-01-09 21:03:44 on dist-test-slave-g0zb"
server_key: "ae44ae6209beb83265450c3784e4f598"
server_key_iv: "d3ffad3ebcffa06f5377de90464afa92"
server_key_version: "encryptionkey@0"
I20250109 21:03:44.451860 19910 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:44.454140 19910 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20250109 21:03:44.454205 19910 log.cc:1198] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20250109 21:03:44.454397 19910 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: Deleting consensus metadata
I20250109 21:03:44.454581 19910 master_runner.cc:315] Copying system tablet from 127.12.126.126:42099
I20250109 21:03:44.455673 19910 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: tablet copy: Beginning tablet copy session from remote peer at address 127.12.126.126:42099
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.126.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.124.keytab.
Entry for principal oryx/127.12.126.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.124.keytab.
I20250109 21:03:44.460294 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.124:37489
--webserver_interface=127.12.126.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.124.keytab
--principal=oryx/127.12.126.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:42099,127.12.126.125:40035,127.12.126.124:37489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:03:44.463117 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.456255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51294 (local address 127.12.126.126:42099)
0109 21:03:44.456389 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.456394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.456599 (+ 205us) server_negotiation.cc:407] Connection header received
0109 21:03:44.456739 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.456742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.456791 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.456889 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:44.457518 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.458052 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.458616 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.458844 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.459882 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:44.459900 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:44.459903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:44.459926 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:44.461404 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.461863 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.461867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.461869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.461927 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.462187 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.462190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.462191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.462385 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:44.462490 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.462772 (+ 282us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.462956 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250109 21:03:44.463207 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.456256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.0.0.1:51294)
0109 21:03:44.456515 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.456527 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:44.456637 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:44.456912 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:44.456915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:44.457107 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:44.457339 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.457350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.458146 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.458149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:44.458506 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.458513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.458626 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.459379 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:44.459400 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:44.459763 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:44.461513 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.461516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.461519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.461753 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.462023 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.462027 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.462031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.462093 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.462485 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:44.462492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:44.462602 (+ 110us) client_negotiation.cc:769] Sending connection context
0109 21:03:44.462717 (+ 115us) client_negotiation.cc:240] Negotiation successful
0109 21:03:44.462939 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
I20250109 21:03:44.464148 19466 tablet_copy_service.cc:140] P 77fb9c5a4bed4855be740888c1397e6c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fed07e4fff2e449a95dd07aa8cf7925f ({username='oryx', principal='oryx/127.12.126.125@KRBTEST.COM'} at 127.0.0.1:51294)
I20250109 21:03:44.464340 19466 tablet_copy_service.cc:161] P 77fb9c5a4bed4855be740888c1397e6c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fed07e4fff2e449a95dd07aa8cf7925f at {username='oryx', principal='oryx/127.12.126.125@KRBTEST.COM'} at 127.0.0.1:51294: session id = fed07e4fff2e449a95dd07aa8cf7925f-00000000000000000000000000000000
I20250109 21:03:44.466387 19466 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c: Tablet Copy: opened 0 blocks and 1 log segments
I20250109 21:03:44.468240 19910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20250109 21:03:44.470440 19910 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: tablet copy: Starting download of 0 data blocks...
I20250109 21:03:44.470619 19910 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: tablet copy: Starting download of 1 WAL segments...
I20250109 21:03:44.472791 19910 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250109 21:03:44.474931 19466 tablet_copy_service.cc:342] P 77fb9c5a4bed4855be740888c1397e6c: Request end of tablet copy session fed07e4fff2e449a95dd07aa8cf7925f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.126.125@KRBTEST.COM'} at 127.0.0.1:51294
I20250109 21:03:44.475055 19466 tablet_copy_service.cc:434] P 77fb9c5a4bed4855be740888c1397e6c: ending tablet copy session fed07e4fff2e449a95dd07aa8cf7925f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fed07e4fff2e449a95dd07aa8cf7925f
I20250109 21:03:44.481930 19910 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:44.484555 20025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:44.485533 20023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:44.488212 20022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:44.489223 19910 server_base.cc:1034] running on GCE node
I20250109 21:03:44.499100 19910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:44.499451 19910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:44.500598 19910 hybrid_clock.cc:648] HybridClock initialized: now 1736456624500591 us; error 36 us; skew 500 ppm
Jan 09 21:03:44 dist-test-slave-g0zb krb5kdc[19401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456624, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:44.502274 19910 init.cc:377] Logged in from keytab as oryx/127.12.126.125@KRBTEST.COM (short username oryx)
I20250109 21:03:44.502718 19910 webserver.cc:458] Webserver started at http://127.12.126.125:37895/ using document root <none> and password file <none>
I20250109 21:03:44.502936 19910 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:44.502970 19910 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:44.507200 19910 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20250109 21:03:44.508248 20032 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:44.508497 19910 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250109 21:03:44.508594 19910 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/wal
uuid: "fed07e4fff2e449a95dd07aa8cf7925f"
format_stamp: "Formatted at 2025-01-09 21:03:44 on dist-test-slave-g0zb"
server_key: "ae44ae6209beb83265450c3784e4f598"
server_key_iv: "d3ffad3ebcffa06f5377de90464afa92"
server_key_version: "encryptionkey@0"
I20250109 21:03:44.508674 19910 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:44.535200 19910 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:44.537534 19910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:44.542264 19910 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.125:40035
I20250109 21:03:44.542311 20084 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.125:40035 every 8 connection(s)
I20250109 21:03:44.543057 19910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-1/data/info.pb
I20250109 21:03:44.544987 20085 sys_catalog.cc:263] Verifying existing consensus state
I20250109 21:03:44.546070 20085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: Bootstrap starting.
I20250109 21:03:44.556821 20085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:03:44.557144 20085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: Bootstrap complete.
I20250109 21:03:44.557791 20085 raft_consensus.cc:357] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } }
I20250109 21:03:44.557950 20085 raft_consensus.cc:738] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fed07e4fff2e449a95dd07aa8cf7925f, State: Initialized, Role: LEARNER
I20250109 21:03:44.558126 20085 consensus_queue.cc:260] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } }
I20250109 21:03:44.558538 20088 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } } }
I20250109 21:03:44.558655 20085 sys_catalog.cc:564] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:44.558739 20088 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [sys.catalog]: This master's current role is: LEARNER
I20250109 21:03:44.561990 20099 catalog_manager.cc:1260] Loaded cluster ID: 4bd8a54b43d446b0b0d221053a3d2adc
I20250109 21:03:44.562077 20099 catalog_manager.cc:1553] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: loading cluster ID for follower catalog manager: success
I20250109 21:03:44.564258 20099 catalog_manager.cc:1575] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: acquiring CA information for follower catalog manager: success
I20250109 21:03:44.565645 20099 catalog_manager.cc:1603] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250109 21:03:44.590096 20021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:44.590335 20021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:44.590384 20021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:44.593289 20021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:44.593341 20021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:44.593362 20021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:44.593385 20021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:44.593407 20021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:03:44.597410 20021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:42099,127.12.126.125:40035,127.12.126.124:37489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.124.keytab
--principal=oryx/127.12.126.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.124:37489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.126.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:44.598403 20021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:44.599182 20021 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:44.606326 20105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:44.607161 20104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:44.607913 20107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:44.608875 20021 server_base.cc:1034] running on GCE node
I20250109 21:03:44.609367 20021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:44.610380 20021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:44.611518 20021 hybrid_clock.cc:648] HybridClock initialized: now 1736456624611506 us; error 24 us; skew 500 ppm
Jan 09 21:03:44 dist-test-slave-g0zb krb5kdc[19401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456624, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:44.614157 20021 init.cc:377] Logged in from keytab as oryx/127.12.126.124@KRBTEST.COM (short username oryx)
I20250109 21:03:44.615022 20021 webserver.cc:458] Webserver started at http://127.12.126.124:39729/ using document root <none> and password file <none>
I20250109 21:03:44.615538 20021 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:44.615587 20021 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:44.615741 20021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:44.617517 20021 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/data/instance:
uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6"
format_stamp: "Formatted at 2025-01-09 21:03:44 on dist-test-slave-g0zb"
server_key: "3e916adc58663515f1b5a484894e3946"
server_key_iv: "e786b13b2c984f6bf2616b48118b02e5"
server_key_version: "encryptionkey@0"
I20250109 21:03:44.617949 20021 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/wal/instance:
uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6"
format_stamp: "Formatted at 2025-01-09 21:03:44 on dist-test-slave-g0zb"
server_key: "3e916adc58663515f1b5a484894e3946"
server_key_iv: "e786b13b2c984f6bf2616b48118b02e5"
server_key_version: "encryptionkey@0"
I20250109 21:03:44.621243 20021 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250109 21:03:44.623320 20114 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:44.624349 20021 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:44.624452 20021 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/wal
uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6"
format_stamp: "Formatted at 2025-01-09 21:03:44 on dist-test-slave-g0zb"
server_key: "3e916adc58663515f1b5a484894e3946"
server_key_iv: "e786b13b2c984f6bf2616b48118b02e5"
server_key_version: "encryptionkey@0"
I20250109 21:03:44.624588 20021 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:44.646943 20021 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:44.649848 20021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:44.649986 20021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:44.655978 20021 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.124:37489
I20250109 21:03:44.656214 20166 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.124:37489 every 8 connection(s)
I20250109 21:03:44.656973 20021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/data/info.pb
I20250109 21:03:44.659512 20167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:44.662750 20167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } has no permanent_uuid. Determining permanent_uuid...
I20250109 21:03:44.667049 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 20021
Jan 09 21:03:44 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456624, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.124@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM
Jan 09 21:03:44 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.126.124@KRBTEST.COM
I20250109 21:03:44.674602 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.664478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51298 (local address 127.12.126.126:42099)
0109 21:03:44.664615 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.664619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.665179 (+ 560us) server_negotiation.cc:407] Connection header received
0109 21:03:44.665874 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.665877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.665943 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.666017 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:44.667344 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.668032 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.668755 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.668897 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.671083 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:44.671100 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:44.671102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:44.671125 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:44.672486 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.673054 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.673057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.673058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.673101 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.673424 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.673426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.673429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.673628 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:44.673743 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.674300 (+ 557us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.674409 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250109 21:03:44.675334 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.664641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.0.0.1:51298)
0109 21:03:44.665049 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.665084 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:44.665706 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:44.666150 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:44.666158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:44.666533 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:44.667192 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.667210 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.668152 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.668157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:44.668639 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.668650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.668888 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.669502 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:44.669524 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:44.670959 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:44.672593 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.672601 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.672615 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.672946 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.673189 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.673192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.673194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.673334 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.673759 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:44.673768 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:44.674068 (+ 300us) client_negotiation.cc:769] Sending connection context
0109 21:03:44.674256 (+ 188us) client_negotiation.cc:240] Negotiation successful
0109 21:03:44.674552 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":107,"threads_started":1}
I20250109 21:03:44.676744 20167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } has no permanent_uuid. Determining permanent_uuid...
I20250109 21:03:44.676880 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.667852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57392 (local address 127.12.126.124:37489)
0109 21:03:44.668135 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.668139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.668159 (+ 20us) server_negotiation.cc:407] Connection header received
0109 21:03:44.668198 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.668202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.668330 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.668452 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:44.669396 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.670146 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.670723 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.670964 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.673453 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:44.673473 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:44.673486 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:44.673506 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:44.675274 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.675689 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.675694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.675696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.675754 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.675992 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.675995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.675996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.676208 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:44.676384 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.676516 (+ 132us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.676649 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":104,"threads_started":1}
Jan 09 21:03:44 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456624, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.124@KRBTEST.COM for oryx/127.12.126.125@KRBTEST.COM
I20250109 21:03:44.683580 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.677264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:40035 (local address 127.0.0.1:53304)
0109 21:03:44.677410 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.677420 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:44.677477 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:44.677865 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:44.677868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:44.678044 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:44.678258 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.678265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.679202 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.679205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:44.679555 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.679562 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.679646 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.680062 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:44.680074 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:44.681123 (+ 1049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:44.682596 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.682599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.682600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.682745 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.682954 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.682956 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.682957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.682995 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.683244 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:44.683246 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:44.683307 (+ 61us) client_negotiation.cc:769] Sending connection context
0109 21:03:44.683378 (+ 71us) client_negotiation.cc:240] Negotiation successful
0109 21:03:44.683468 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20250109 21:03:44.683633 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.677386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53304 (local address 127.12.126.125:40035)
0109 21:03:44.677656 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.677659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.677671 (+ 12us) server_negotiation.cc:407] Connection header received
0109 21:03:44.677706 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.677708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.677749 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.677833 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:44.678352 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.679113 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.679638 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.679763 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.681228 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:44.681245 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:44.681248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:44.681268 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:44.682505 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.682847 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.682850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.682851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.682891 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.683054 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.683055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.683056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.683181 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:44.683248 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.683417 (+ 169us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.683517 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":94,"threads_started":1}
I20250109 21:03:44.684314 20167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } has no permanent_uuid. Determining permanent_uuid...
Jan 09 21:03:44 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456624, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.124@KRBTEST.COM for oryx/127.12.126.124@KRBTEST.COM
I20250109 21:03:44.690429 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.684745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:37489 (local address 127.0.0.1:57408)
0109 21:03:44.684844 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.684855 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:44.684913 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:44.685082 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:44.685084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:44.685195 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:44.685343 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.685347 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.685924 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.685926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:44.686249 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.686254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.686333 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.686743 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:44.686756 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:44.687853 (+ 1097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:44.689323 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.689326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.689328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.689473 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.689673 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.689676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.689678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.689719 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.689988 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:44.689991 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:44.690055 (+ 64us) client_negotiation.cc:769] Sending connection context
0109 21:03:44.690142 (+ 87us) client_negotiation.cc:240] Negotiation successful
0109 21:03:44.690277 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20250109 21:03:44.690431 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.684843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.12.126.124:37489)
0109 21:03:44.684934 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.684937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.684948 (+ 11us) server_negotiation.cc:407] Connection header received
0109 21:03:44.684973 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.684975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.685007 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.685067 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:44.685423 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.685855 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.686329 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.686443 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.687960 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:44.687972 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:44.687975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:44.687993 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:44.689230 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.689552 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.689555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.689558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.689601 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.689790 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.689793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.689794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.689912 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:44.689992 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.690197 (+ 205us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.690329 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":33}
I20250109 21:03:44.693485 20167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: Bootstrap starting.
I20250109 21:03:44.695693 20167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:44.696396 20167 log.cc:826] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:44.698228 20167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: No bootstrap required, opened a new log
I20250109 21:03:44.700835 20167 raft_consensus.cc:357] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } }
I20250109 21:03:44.701031 20167 raft_consensus.cc:383] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:44.701099 20167 raft_consensus.cc:738] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a956bf10bf8d4c388a73b2a1dc6a1cf6, State: Initialized, Role: FOLLOWER
I20250109 21:03:44.701584 20167 consensus_queue.cc:260] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } }
I20250109 21:03:44.702014 20173 sys_catalog.cc:455] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } } }
I20250109 21:03:44.702119 20173 sys_catalog.cc:458] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:03:44.702690 20167 sys_catalog.cc:564] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: configured and running, proceeding with master startup.
W20250109 21:03:44.707240 20184 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:03:44.707330 20184 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 21:03:44.710506 20021 master_runner.cc:428] Detected that this master a956bf10bf8d4c388a73b2a1dc6a1cf6 is joining an existing cluster
I20250109 21:03:44.710564 20021 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.126.124:37489
I20250109 21:03:44.735311 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.730645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.0.0.1:51304)
0109 21:03:44.730922 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.730935 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:44.731007 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:44.732160 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:44.732165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:44.732182 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:44.732440 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.732453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.733889 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.733892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:44.734873 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.734884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.734982 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.735053 (+ 71us) client_negotiation.cc:769] Sending connection context
0109 21:03:44.735097 (+ 44us) client_negotiation.cc:240] Negotiation successful
0109 21:03:44.735163 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20250109 21:03:44.736160 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.712748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51302 (local address 127.12.126.126:42099)
0109 21:03:44.712878 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.712884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.713118 (+ 234us) server_negotiation.cc:407] Connection header received
0109 21:03:44.713316 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.713321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.713378 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.713452 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:44.727901 (+ 14449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.728641 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.729570 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.729759 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.731435 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:44.731450 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:44.731453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:44.731488 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:44.734124 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.734928 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.734932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.734934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.734988 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.735266 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.735269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.735271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.735447 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:44.735537 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.735884 (+ 347us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.735992 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250109 21:03:44.736501 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.730978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51304 (local address 127.12.126.126:42099)
0109 21:03:44.731862 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.731869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.731881 (+ 12us) server_negotiation.cc:407] Connection header received
0109 21:03:44.731978 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.731984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.732044 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.732855 (+ 811us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:44.732948 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.733774 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.735754 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.736208 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.736285 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.736333 (+ 48us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.736384 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":131,"threads_started":1}
I20250109 21:03:44.736913 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.712652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.0.0.1:51302)
0109 21:03:44.713026 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.713041 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:44.713148 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:44.713473 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:44.713476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:44.727201 (+ 13725us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:44.727643 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.727658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.728788 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.728792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:44.729416 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.729424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.729888 (+ 464us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.730548 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:44.730564 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:44.731304 (+ 740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:44.734560 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.734564 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.734567 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.734786 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.735106 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.735111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.735116 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.735171 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.735589 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:44.735592 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:44.735681 (+ 89us) client_negotiation.cc:769] Sending connection context
0109 21:03:44.736560 (+ 879us) client_negotiation.cc:240] Negotiation successful
0109 21:03:44.736696 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":101,"threads_started":1}
I20250109 21:03:44.737335 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.727271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.12.126.125:40035)
0109 21:03:44.727447 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.727453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.728124 (+ 671us) server_negotiation.cc:407] Connection header received
0109 21:03:44.728329 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.728332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.728388 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.728455 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:44.729642 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.730372 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.731242 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.731444 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.732575 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:44.732596 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:44.732599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:44.732630 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:44.735251 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.735948 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.735952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.735957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.736008 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.736266 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.736270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.736275 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.736462 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:44.736548 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.737019 (+ 471us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.737155 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20250109 21:03:44.737701 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.727579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:40035 (local address 127.0.0.1:53318)
0109 21:03:44.728046 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.728061 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:44.728197 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:44.728967 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:44.728970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:44.729189 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:44.729509 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.729518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.730526 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.730529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:44.731080 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.731095 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.731559 (+ 464us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.732111 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:44.732126 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:44.732445 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:44.735640 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.735643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.735645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.735847 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.736104 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.736107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.736109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.736155 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.736587 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:44.736590 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:44.736678 (+ 88us) client_negotiation.cc:769] Sending connection context
0109 21:03:44.736775 (+ 97us) client_negotiation.cc:240] Negotiation successful
0109 21:03:44.736899 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":149,"threads_started":1}
I20250109 21:03:44.738763 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.713018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57420 (local address 127.12.126.124:37489)
0109 21:03:44.713116 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.713120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.713181 (+ 61us) server_negotiation.cc:407] Connection header received
0109 21:03:44.713358 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.713361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.713403 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.713470 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:44.727771 (+ 14301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.728494 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.730448 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.730622 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.731740 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:44.731754 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:44.731757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:44.731778 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:44.733790 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.737726 (+ 3936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.737732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.737736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.737789 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.738025 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.738031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.738035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.738184 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:44.738263 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.738527 (+ 264us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.738641 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20250109 21:03:44.739446 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.712704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:37489 (local address 127.0.0.1:57420)
0109 21:03:44.713116 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.713129 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:44.713240 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:44.713567 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:44.713570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:44.727161 (+ 13591us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:44.727594 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.727608 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.728722 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.728725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:44.730315 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.730325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.730715 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.731256 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:44.731272 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:44.731625 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:44.734285 (+ 2660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.734290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.734293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.737622 (+ 3329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.737883 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.737886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.737888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.737931 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.738303 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:44.738305 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:44.738370 (+ 65us) client_negotiation.cc:769] Sending connection context
0109 21:03:44.739021 (+ 651us) client_negotiation.cc:240] Negotiation successful
0109 21:03:44.739136 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"spinlock_wait_cycles":10880,"thread_start_us":82,"threads_started":1}
I20250109 21:03:44.742329 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.737903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:40035 (local address 127.0.0.1:53334)
0109 21:03:44.739181 (+ 1278us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.739195 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:44.739278 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:44.739862 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:44.739866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:44.739888 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:44.740107 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.740121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.741067 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.741071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:44.741959 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.741969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.742075 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.742092 (+ 17us) client_negotiation.cc:769] Sending connection context
0109 21:03:44.742141 (+ 49us) client_negotiation.cc:240] Negotiation successful
0109 21:03:44.742213 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1207}
I20250109 21:03:44.742982 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.738113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53334 (local address 127.12.126.125:40035)
0109 21:03:44.739431 (+ 1318us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.739435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.739448 (+ 13us) server_negotiation.cc:407] Connection header received
0109 21:03:44.739487 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.739493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.739561 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.739663 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:44.740252 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.740952 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.742100 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.742578 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.742737 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.742789 (+ 52us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.742874 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1242}
Jan 09 21:03:44 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456623, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.126@KRBTEST.COM for oryx/127.12.126.124@KRBTEST.COM
I20250109 21:03:44.749189 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.741806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57424 (local address 127.12.126.124:37489)
0109 21:03:44.741923 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.741927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.742138 (+ 211us) server_negotiation.cc:407] Connection header received
0109 21:03:44.742370 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.742373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.742425 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.742474 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:44.743169 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.743651 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.744373 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.744554 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.746095 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:44.746113 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:44.746117 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:44.746140 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:44.747779 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.748189 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.748194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.748196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.748249 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.748505 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.748509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.748510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.748642 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:44.748725 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.748946 (+ 221us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.749041 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250109 21:03:44.749197 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.741746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:37489 (local address 127.0.0.1:57424)
0109 21:03:44.742071 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.742085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:44.742213 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:44.742609 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:44.742612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:44.742825 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:44.743033 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.743042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.743792 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.743795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:44.744247 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.744257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.744364 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.744841 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:44.744859 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:44.745979 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:44.747878 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.747881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.747883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.748100 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.748351 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.748357 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.748359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.748415 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.748725 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:44.748729 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:44.748805 (+ 76us) client_negotiation.cc:769] Sending connection context
0109 21:03:44.748892 (+ 87us) client_negotiation.cc:240] Negotiation successful
0109 21:03:44.749020 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20250109 21:03:44.750082 19445 catalog_manager.cc:7010] Initiating ChangeConfig request to add master 127.12.126.124:37489: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6"
member_type: NON_VOTER
last_known_addr {
host: "127.12.126.124"
port: 37489
}
attrs {
promote: true
}
}
dest_uuid: "77fb9c5a4bed4855be740888c1397e6c"
cas_config_opid_index: 5
I20250109 21:03:44.750638 19445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } }
I20250109 21:03:44.751646 20007 raft_consensus.cc:2949] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a956bf10bf8d4c388a73b2a1dc6a1cf6 (127.12.126.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } } }
I20250109 21:03:44.752252 20141 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:44.752828 20007 catalog_manager.cc:7014] Successfully completed master ChangeConfig request to add master 127.12.126.124:37489
I20250109 21:03:44.753002 20007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } } }
I20250109 21:03:44.753103 20007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:44.754045 20141 raft_consensus.cc:1270] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 1 FOLLOWER]: Refusing update from remote peer 77fb9c5a4bed4855be740888c1397e6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250109 21:03:44.754570 20059 raft_consensus.cc:1270] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 1 LEARNER]: Refusing update from remote peer 77fb9c5a4bed4855be740888c1397e6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250109 21:03:44.754609 20007 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:44.755118 20006 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250109 21:03:44.755522 20021 master.cc:537] Master@127.12.126.124:37489 shutting down...
W20250109 21:03:44.757287 19446 master.cc:655] Remote error: unable to get registration information for peer a956bf10bf8d4c388a73b2a1dc6a1cf6: Service unavailable: service kudu.master.MasterService not registered on Master
I20250109 21:03:44.757526 20059 raft_consensus.cc:2949] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a956bf10bf8d4c388a73b2a1dc6a1cf6 (127.12.126.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } } }
I20250109 21:03:44.759287 20088 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77fb9c5a4bed4855be740888c1397e6c. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } } }
I20250109 21:03:44.759308 20141 raft_consensus.cc:867] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } }
I20250109 21:03:44.759385 20088 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [sys.catalog]: This master's current role is: LEARNER
W20250109 21:03:44.759403 20141 raft_consensus.cc:1574] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } }
I20250109 21:03:44.759460 20088 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } } }
I20250109 21:03:44.759538 20088 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [sys.catalog]: This master's current role is: LEARNER
I20250109 21:03:44.761675 20007 raft_consensus.cc:1059] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c: attempting to promote NON_VOTER fed07e4fff2e449a95dd07aa8cf7925f to VOTER
I20250109 21:03:44.761423 20141 raft_consensus.cc:2949] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fed07e4fff2e449a95dd07aa8cf7925f (127.12.126.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } } }
I20250109 21:03:44.762033 20007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } }
I20250109 21:03:44.763959 20059 raft_consensus.cc:1270] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 1 LEARNER]: Refusing update from remote peer 77fb9c5a4bed4855be740888c1397e6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250109 21:03:44.763993 20196 mvcc.cc:204] Tried to move back new op lower bound from 7112526330032156672 to 7112526329888501760. Current Snapshot: MvccSnapshot[applied={T|T < 7112526329920270336}]
I20250109 21:03:44.763978 20199 sys_catalog.cc:455] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } } }
I20250109 21:03:44.764149 20199 sys_catalog.cc:458] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20250109 21:03:44.764447 20006 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250109 21:03:44.764094 20202 sys_catalog.cc:455] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } } }
I20250109 21:03:44.766106 20006 raft_consensus.cc:2949] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fed07e4fff2e449a95dd07aa8cf7925f (127.12.126.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } } }
W20250109 21:03:44.766292 19430 consensus_peers.cc:487] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c -> Peer a956bf10bf8d4c388a73b2a1dc6a1cf6 (127.12.126.124:37489): Couldn't send request to peer a956bf10bf8d4c388a73b2a1dc6a1cf6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20250109 21:03:44.767551 20006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } } }
I20250109 21:03:44.767644 20006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:44.767671 20007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } } }
I20250109 21:03:44.767815 20007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:44.765311 20202 sys_catalog.cc:458] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20250109 21:03:44.768285 20059 raft_consensus.cc:2949] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fed07e4fff2e449a95dd07aa8cf7925f (127.12.126.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } } }
I20250109 21:03:44.769590 20197 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } } }
I20250109 21:03:44.769711 20197 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:03:44.769953 19637 heartbeater.cc:502] Master 127.12.126.126:42099 was elected leader, sending a full tablet report...
I20250109 21:03:44.767493 20021 raft_consensus.cc:2238] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20250109 21:03:44.765836 20173 sys_catalog.cc:455] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77fb9c5a4bed4855be740888c1397e6c. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: NON_VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: true } } }
I20250109 21:03:44.770092 20173 sys_catalog.cc:458] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20250109 21:03:44.770264 20021 raft_consensus.cc:2267] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20250109 21:03:44.770352 20021 tablet_replica.cc:331] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: stopping tablet replica
W20250109 21:03:44.770718 19446 master.cc:655] Remote error: unable to get registration information for peer a956bf10bf8d4c388a73b2a1dc6a1cf6: Service unavailable: service kudu.master.MasterService not registered on Master
I20250109 21:03:44.782218 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.781939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:37489 (local address 127.0.0.1:57432)
0109 21:03:44.782064 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.782128 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:37489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20250109 21:03:44.782439 19446 master.cc:655] Network error: unable to get registration information for peer a956bf10bf8d4c388a73b2a1dc6a1cf6: Client connection negotiation failed: client connection to 127.12.126.124:37489: connect: Connection refused (error 111)
I20250109 21:03:44.785969 20021 master.cc:559] Master@127.12.126.124:37489 shutdown complete.
I20250109 21:03:44.786016 20021 master_runner.cc:305] Clearing existing system tablet
I20250109 21:03:44.786219 20021 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:44.786269 20021 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:44.787395 20021 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20250109 21:03:44.788169 20205 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:44.788898 20021 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:44.789001 20021 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250109 21:03:44.789031 20021 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/wal
uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6"
format_stamp: "Formatted at 2025-01-09 21:03:44 on dist-test-slave-g0zb"
server_key: "3e916adc58663515f1b5a484894e3946"
server_key_iv: "e786b13b2c984f6bf2616b48118b02e5"
server_key_version: "encryptionkey@0"
I20250109 21:03:44.790437 20021 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:44.792161 20021 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20250109 21:03:44.792227 20021 log.cc:1198] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20250109 21:03:44.792491 20021 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: Deleting consensus metadata
I20250109 21:03:44.792644 20021 master_runner.cc:315] Copying system tablet from 127.12.126.126:42099
I20250109 21:03:44.793661 20021 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: tablet copy: Beginning tablet copy session from remote peer at address 127.12.126.126:42099
I20250109 21:03:44.800547 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.794243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51312 (local address 127.12.126.126:42099)
0109 21:03:44.794372 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.794378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.794523 (+ 145us) server_negotiation.cc:407] Connection header received
0109 21:03:44.794653 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.794656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.794701 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.794773 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:44.795316 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.795822 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.796410 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.796609 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.797481 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:44.797498 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:44.797501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:44.797524 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:44.798988 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.799348 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.799352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.799354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.799399 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:44.799720 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:44.799723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:44.799724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:44.799900 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:44.799992 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.800263 (+ 271us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.800388 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250109 21:03:44.800984 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.794186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.0.0.1:51312)
0109 21:03:44.794439 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.794450 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:44.794560 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:44.794819 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:44.794822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:44.795014 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:44.795196 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.795201 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.795928 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.795934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:44.796305 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.796312 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.796415 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.797060 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:44.797073 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:44.797363 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:44.799092 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.799096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.799098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.799264 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.799496 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:44.799499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:44.799502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:44.799563 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:44.799991 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:44.799994 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:44.800079 (+ 85us) client_negotiation.cc:769] Sending connection context
0109 21:03:44.800689 (+ 610us) client_negotiation.cc:240] Negotiation successful
0109 21:03:44.800829 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":105,"threads_started":1}
I20250109 21:03:44.801585 19466 tablet_copy_service.cc:140] P 77fb9c5a4bed4855be740888c1397e6c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a956bf10bf8d4c388a73b2a1dc6a1cf6 ({username='oryx', principal='oryx/127.12.126.124@KRBTEST.COM'} at 127.0.0.1:51312)
I20250109 21:03:44.801663 19466 tablet_copy_service.cc:161] P 77fb9c5a4bed4855be740888c1397e6c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a956bf10bf8d4c388a73b2a1dc6a1cf6 at {username='oryx', principal='oryx/127.12.126.124@KRBTEST.COM'} at 127.0.0.1:51312: session id = a956bf10bf8d4c388a73b2a1dc6a1cf6-00000000000000000000000000000000
I20250109 21:03:44.802405 19466 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c: Tablet Copy: opened 0 blocks and 1 log segments
I20250109 21:03:44.802994 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.802737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:37489 (local address 127.0.0.1:57442)
0109 21:03:44.802879 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.802932 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:37489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20250109 21:03:44.803182 19446 master.cc:655] Network error: unable to get registration information for peer a956bf10bf8d4c388a73b2a1dc6a1cf6: Client connection negotiation failed: client connection to 127.12.126.124:37489: connect: Connection refused (error 111)
I20250109 21:03:44.803990 20021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20250109 21:03:44.805640 20021 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: tablet copy: Starting download of 0 data blocks...
I20250109 21:03:44.805796 20021 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: tablet copy: Starting download of 1 WAL segments...
I20250109 21:03:44.807384 20021 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250109 21:03:44.808782 19466 tablet_copy_service.cc:342] P 77fb9c5a4bed4855be740888c1397e6c: Request end of tablet copy session a956bf10bf8d4c388a73b2a1dc6a1cf6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.126.124@KRBTEST.COM'} at 127.0.0.1:51312
I20250109 21:03:44.808856 19466 tablet_copy_service.cc:434] P 77fb9c5a4bed4855be740888c1397e6c: ending tablet copy session a956bf10bf8d4c388a73b2a1dc6a1cf6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a956bf10bf8d4c388a73b2a1dc6a1cf6
I20250109 21:03:44.814949 20021 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250109 21:03:44.817500 20021 server_base.cc:1034] running on GCE node
W20250109 21:03:44.818415 20212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:44.819161 20215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:44.820097 20213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:44.820389 20021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:44.820698 20021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:44.821835 20021 hybrid_clock.cc:648] HybridClock initialized: now 1736456624821817 us; error 38 us; skew 500 ppm
Jan 09 21:03:44 dist-test-slave-g0zb krb5kdc[19401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456624, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:44.823468 20021 init.cc:377] Logged in from keytab as oryx/127.12.126.124@KRBTEST.COM (short username oryx)
I20250109 21:03:44.823923 20021 webserver.cc:458] Webserver started at http://127.12.126.124:37561/ using document root <none> and password file <none>
I20250109 21:03:44.824115 20021 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:44.824153 20021 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:44.825435 20021 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250109 21:03:44.826362 20222 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:44.826620 20021 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250109 21:03:44.826694 20021 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/wal
uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6"
format_stamp: "Formatted at 2025-01-09 21:03:44 on dist-test-slave-g0zb"
server_key: "3e916adc58663515f1b5a484894e3946"
server_key_iv: "e786b13b2c984f6bf2616b48118b02e5"
server_key_version: "encryptionkey@0"
I20250109 21:03:44.826781 20021 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:44.835157 20021 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:44.837265 20021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:44.838972 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.838674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:37489 (local address 127.0.0.1:57454)
0109 21:03:44.838833 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.838890 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:37489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20250109 21:03:44.839169 19446 master.cc:655] Network error: unable to get registration information for peer a956bf10bf8d4c388a73b2a1dc6a1cf6: Client connection negotiation failed: client connection to 127.12.126.124:37489: connect: Connection refused (error 111)
I20250109 21:03:44.842063 20021 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.124:37489
I20250109 21:03:44.842083 20274 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.124:37489 every 8 connection(s)
I20250109 21:03:44.842653 20021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/master-2/data/info.pb
I20250109 21:03:44.844091 20275 sys_catalog.cc:263] Verifying existing consensus state
I20250109 21:03:44.844832 20275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: Bootstrap starting.
I20250109 21:03:44.851097 20275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:03:44.851352 20275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: Bootstrap complete.
I20250109 21:03:44.851787 20275 raft_consensus.cc:357] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } }
I20250109 21:03:44.851889 20275 raft_consensus.cc:738] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a956bf10bf8d4c388a73b2a1dc6a1cf6, State: Initialized, Role: LEARNER
I20250109 21:03:44.852002 20275 consensus_queue.cc:260] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } }
I20250109 21:03:44.852275 20278 sys_catalog.cc:455] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } } }
I20250109 21:03:44.852378 20278 sys_catalog.cc:458] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: This master's current role is: LEARNER
I20250109 21:03:44.852429 20275 sys_catalog.cc:564] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:44.855278 20289 catalog_manager.cc:1260] Loaded cluster ID: 4bd8a54b43d446b0b0d221053a3d2adc
I20250109 21:03:44.855396 20289 catalog_manager.cc:1553] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: loading cluster ID for follower catalog manager: success
I20250109 21:03:44.857592 20289 catalog_manager.cc:1575] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: acquiring CA information for follower catalog manager: success
I20250109 21:03:44.859040 20289 catalog_manager.cc:1603] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 21:03:44.909313 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.906634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:37489 (local address 127.0.0.1:57462)
0109 21:03:44.906772 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:44.906786 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:44.906933 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:44.907338 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:44.907340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:44.907357 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:44.907549 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.907553 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.908388 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.908390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:44.909007 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:44.909016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.909106 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.909123 (+ 17us) client_negotiation.cc:769] Sending connection context
0109 21:03:44.909160 (+ 37us) client_negotiation.cc:240] Negotiation successful
0109 21:03:44.909205 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20250109 21:03:44.909919 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:44.906761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57462 (local address 127.12.126.124:37489)
0109 21:03:44.907094 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:44.907097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:44.907110 (+ 13us) server_negotiation.cc:407] Connection header received
0109 21:03:44.907154 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:44.907156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:44.907209 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:44.907325 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:44.907632 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.908316 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:44.909100 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:44.909544 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:44.909704 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:44.909749 (+ 45us) server_negotiation.cc:299] Negotiation successful
0109 21:03:44.909812 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":130,"threads_started":1}
I20250109 21:03:44.975553 19771 heartbeater.cc:502] Master 127.12.126.126:42099 was elected leader, sending a full tablet report...
I20250109 21:03:45.174278 19905 heartbeater.cc:502] Master 127.12.126.126:42099 was elected leader, sending a full tablet report...
I20250109 21:03:45.357543 20241 raft_consensus.cc:1212] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250109 21:03:45.357971 20291 sys_catalog.cc:455] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77fb9c5a4bed4855be740888c1397e6c. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } } }
I20250109 21:03:45.358119 20291 sys_catalog.cc:458] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: This master's current role is: LEARNER
I20250109 21:03:45.886603 20007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: NON_VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: true } } }
I20250109 21:03:45.886824 20007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:45.887676 20293 raft_consensus.cc:1059] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c: attempting to promote NON_VOTER a956bf10bf8d4c388a73b2a1dc6a1cf6 to VOTER
I20250109 21:03:45.888024 20293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: false } }
I20250109 21:03:45.889231 20241 raft_consensus.cc:1270] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 1 LEARNER]: Refusing update from remote peer 77fb9c5a4bed4855be740888c1397e6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250109 21:03:45.889261 20059 raft_consensus.cc:1270] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 1 FOLLOWER]: Refusing update from remote peer 77fb9c5a4bed4855be740888c1397e6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250109 21:03:45.889664 20292 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250109 21:03:45.889918 20007 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250109 21:03:45.891754 20293 raft_consensus.cc:2949] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a956bf10bf8d4c388a73b2a1dc6a1cf6 (127.12.126.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: false } } }
I20250109 21:03:45.892191 20059 raft_consensus.cc:2949] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a956bf10bf8d4c388a73b2a1dc6a1cf6 (127.12.126.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: false } } }
I20250109 21:03:45.892315 20241 raft_consensus.cc:2949] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a956bf10bf8d4c388a73b2a1dc6a1cf6 (127.12.126.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: false } } }
I20250109 21:03:45.893203 20007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: false } } }
I20250109 21:03:45.893249 20292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: false } } }
I20250109 21:03:45.893301 20007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:45.893313 20292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:45.893491 20292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: false } } }
I20250109 21:03:45.893574 20292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:45.893806 20298 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: false } } }
I20250109 21:03:45.893982 20298 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:03:45.893965 20299 sys_catalog.cc:455] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "77fb9c5a4bed4855be740888c1397e6c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: false } } }
I20250109 21:03:45.894105 20299 sys_catalog.cc:458] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:03:46.831378 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:46.824978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.12.126.125:40035)
0109 21:03:46.825219 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:46.825222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:46.825252 (+ 30us) server_negotiation.cc:407] Connection header received
0109 21:03:46.825384 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:46.825386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:46.825435 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:46.825523 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:46.826286 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.826845 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:46.827444 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.827613 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:46.828795 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:46.828810 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:46.828813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:46.828836 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:46.830155 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:46.830489 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:46.830492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:46.830493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:46.830528 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:46.830674 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:46.830676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:46.830677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:46.830830 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:46.830923 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:46.831079 (+ 156us) server_negotiation.cc:299] Negotiation successful
0109 21:03:46.831171 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":111,"threads_started":1}
I20250109 21:03:46.834407 20302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:46.832138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.0.0.1:51322)
0109 21:03:46.832358 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:46.832371 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:46.832441 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:46.832673 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:46.832675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:46.832685 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:46.832856 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:46.832860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.833547 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:46.833549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:46.834112 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:46.834118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.834206 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:46.834221 (+ 15us) client_negotiation.cc:769] Sending connection context
0109 21:03:46.834258 (+ 37us) client_negotiation.cc:240] Negotiation successful
0109 21:03:46.834304 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":65,"threads_started":1}
I20250109 21:03:46.834858 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:46.832256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51322 (local address 127.12.126.126:42099)
0109 21:03:46.832490 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:46.832494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:46.832506 (+ 12us) server_negotiation.cc:407] Connection header received
0109 21:03:46.832548 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:46.832550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:46.832591 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:46.832664 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:46.832945 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.833459 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:46.834200 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.834582 (+ 382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:46.834619 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:46.834668 (+ 49us) server_negotiation.cc:299] Negotiation successful
0109 21:03:46.834715 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":72,"threads_started":1}
I20250109 21:03:46.837961 20302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:46.835984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:40035 (local address 127.0.0.1:53348)
0109 21:03:46.836121 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:46.836137 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:46.836199 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:46.836400 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:46.836402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:46.836416 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:46.836560 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:46.836564 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.837163 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:46.837165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:46.837704 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:46.837711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.837791 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:46.837804 (+ 13us) client_negotiation.cc:769] Sending connection context
0109 21:03:46.837839 (+ 35us) client_negotiation.cc:240] Negotiation successful
0109 21:03:46.837880 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20250109 21:03:46.838348 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:46.836055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53348 (local address 127.12.126.125:40035)
0109 21:03:46.836151 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:46.836154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:46.836179 (+ 25us) server_negotiation.cc:407] Connection header received
0109 21:03:46.836288 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:46.836290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:46.836325 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:46.836380 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:46.836637 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.837087 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:46.837793 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.838147 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:46.838181 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:46.838221 (+ 40us) server_negotiation.cc:299] Negotiation successful
0109 21:03:46.838275 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250109 21:03:46.841724 20302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:46.839318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:37489 (local address 127.0.0.1:57468)
0109 21:03:46.839432 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:46.839444 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:46.839502 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:46.839954 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:46.839958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:46.839975 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:46.840120 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:46.840124 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.840843 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:46.840847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:46.841375 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:46.841383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.841487 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:46.841504 (+ 17us) client_negotiation.cc:769] Sending connection context
0109 21:03:46.841564 (+ 60us) client_negotiation.cc:240] Negotiation successful
0109 21:03:46.841617 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20250109 21:03:46.842216 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:46.839495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.12.126.124:37489)
0109 21:03:46.839750 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:46.839754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:46.839766 (+ 12us) server_negotiation.cc:407] Connection header received
0109 21:03:46.839811 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:46.839814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:46.839854 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:46.839934 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:46.840230 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.840738 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:46.841489 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.841942 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:46.841974 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:46.842017 (+ 43us) server_negotiation.cc:299] Negotiation successful
0109 21:03:46.842062 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20250109 21:03:46.850253 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:46.844211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.12.126.124:37489)
0109 21:03:46.844335 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:46.844339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:46.844353 (+ 14us) server_negotiation.cc:407] Connection header received
0109 21:03:46.844414 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:46.844417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:46.844458 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:46.844535 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:46.845239 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.845692 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:46.846259 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.846451 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:46.847637 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:46.847653 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:46.847657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:46.847683 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:46.849119 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:46.849454 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:46.849462 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:46.849463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:46.849499 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:46.849652 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:46.849653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:46.849654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:46.849780 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:46.849878 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:46.850016 (+ 138us) server_negotiation.cc:299] Negotiation successful
0109 21:03:46.850111 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250109 21:03:46.853338 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:46.850989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.0.0.1:51326)
0109 21:03:46.851232 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:46.851244 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:46.851317 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:46.851529 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:46.851531 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:46.851542 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:46.851709 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:46.851714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.852384 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:46.852386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:46.853038 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:46.853047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.853137 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:46.853152 (+ 15us) client_negotiation.cc:769] Sending connection context
0109 21:03:46.853189 (+ 37us) client_negotiation.cc:240] Negotiation successful
0109 21:03:46.853234 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":87,"threads_started":1}
I20250109 21:03:46.853772 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:46.851117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.12.126.126:42099)
0109 21:03:46.851283 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:46.851286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:46.851300 (+ 14us) server_negotiation.cc:407] Connection header received
0109 21:03:46.851402 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:46.851404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:46.851448 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:46.851513 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:46.851788 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.852303 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:46.853140 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.853534 (+ 394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:46.853582 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:46.853634 (+ 52us) server_negotiation.cc:299] Negotiation successful
0109 21:03:46.853681 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20250109 21:03:46.857009 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:46.854966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:40035 (local address 127.0.0.1:53350)
0109 21:03:46.855112 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:46.855127 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:46.855210 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:46.855412 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:46.855414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:46.855427 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:46.855576 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:46.855580 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.856188 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:46.856190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:46.856748 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:46.856754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.856838 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:46.856851 (+ 13us) client_negotiation.cc:769] Sending connection context
0109 21:03:46.856887 (+ 36us) client_negotiation.cc:240] Negotiation successful
0109 21:03:46.856925 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20250109 21:03:46.857393 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:46.855111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53350 (local address 127.12.126.125:40035)
0109 21:03:46.855238 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:46.855242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:46.855253 (+ 11us) server_negotiation.cc:407] Connection header received
0109 21:03:46.855297 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:46.855299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:46.855337 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:46.855400 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:46.855651 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.856111 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:46.856831 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.857178 (+ 347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:46.857214 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:46.857263 (+ 49us) server_negotiation.cc:299] Negotiation successful
0109 21:03:46.857309 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250109 21:03:46.861179 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:46.858464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:37489 (local address 127.0.0.1:57482)
0109 21:03:46.858948 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:46.858964 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:46.859045 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:46.859258 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:46.859261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:46.859278 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:46.859449 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:46.859456 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.860176 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:46.860179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:46.860820 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:46.860828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.860925 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:46.860942 (+ 17us) client_negotiation.cc:769] Sending connection context
0109 21:03:46.860995 (+ 53us) client_negotiation.cc:240] Negotiation successful
0109 21:03:46.861049 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20250109 21:03:46.861548 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:46.858530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.12.126.124:37489)
0109 21:03:46.858626 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:46.858629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:46.859020 (+ 391us) server_negotiation.cc:407] Connection header received
0109 21:03:46.859124 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:46.859127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:46.859174 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:46.859236 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:46.859549 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.860079 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:46.860932 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:46.861376 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:46.861405 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:46.861441 (+ 36us) server_negotiation.cc:299] Negotiation successful
0109 21:03:46.861479 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250109 21:03:46.862869 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 19508
I20250109 21:03:46.871729 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:35925
--local_ip_for_outbound_sockets=127.12.126.65
--tserver_master_addrs=127.12.126.126:42099
--webserver_port=46061
--webserver_interface=127.12.126.65
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.65.keytab
--principal=oryx/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.126.126:42099,127.12.126.125:40035
--tserver_master_addrs=127.12.126.126:42099,127.12.126.125:40035,127.12.126.124:37489 with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:46.984359 20306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:46.984627 20306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:46.984691 20306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:46.987773 20306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:46.987839 20306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:46.987942 20306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:03:46.992187 20306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.65.keytab
--principal=oryx/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:35925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=46061
--tserver_master_addrs=127.12.126.126:42099,127.12.126.125:40035,127.12.126.124:37489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:46.993369 20306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:46.994233 20306 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:47.003093 20311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:47.003942 20312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:47.004628 20314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:47.005468 20306 server_base.cc:1034] running on GCE node
I20250109 21:03:47.005853 20306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:47.006407 20306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:47.007568 20306 hybrid_clock.cc:648] HybridClock initialized: now 1736456627007545 us; error 42 us; skew 500 ppm
Jan 09 21:03:47 dist-test-slave-g0zb krb5kdc[19401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456627, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:47.010398 20306 init.cc:377] Logged in from keytab as oryx/127.12.126.65@KRBTEST.COM (short username oryx)
I20250109 21:03:47.011372 20306 webserver.cc:458] Webserver started at http://127.12.126.65:46061/ using document root <none> and password file <none>
I20250109 21:03:47.011925 20306 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:47.011997 20306 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:47.015509 20306 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:03:47.017418 20321 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:47.018494 20306 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:47.018599 20306 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "ffe52c1a4d014d29a2a57c3120d158b1"
format_stamp: "Formatted at 2025-01-09 21:03:43 on dist-test-slave-g0zb"
server_key: "bc0641937987920803b6e969705111c2"
server_key_iv: "3f3e942cab674f9e896250bf2a582cd8"
server_key_version: "encryptionkey@0"
I20250109 21:03:47.019042 20306 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:47.039094 20306 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:47.042083 20306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:47.042280 20306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:47.042846 20306 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:47.043706 20306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:47.043753 20306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:47.043809 20306 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:47.043839 20306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:47.052096 20306 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:35925
I20250109 21:03:47.052119 20434 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:35925 every 8 connection(s)
I20250109 21:03:47.053124 20306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:03:47.057574 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 20306
I20250109 21:03:47.057775 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 19642
Jan 09 21:03:47 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456627, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.124@KRBTEST.COM
Jan 09 21:03:47 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456627, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM
Jan 09 21:03:47 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456627, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.125@KRBTEST.COM
I20250109 21:03:47.067885 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:45203
--local_ip_for_outbound_sockets=127.12.126.66
--tserver_master_addrs=127.12.126.126:42099
--webserver_port=34881
--webserver_interface=127.12.126.66
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.66.keytab
--principal=oryx/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.126.126:42099,127.12.126.125:40035
--tserver_master_addrs=127.12.126.126:42099,127.12.126.125:40035,127.12.126.124:37489 with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:03:47.068626 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.054865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:36207 (local address 127.12.126.124:37489)
0109 21:03:47.055004 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.055009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.055914 (+ 905us) server_negotiation.cc:407] Connection header received
0109 21:03:47.056947 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.056950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.057013 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.057083 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:47.058840 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.059619 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.060517 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.060694 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.063968 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:47.063989 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:47.063991 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:47.064017 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:47.065726 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.066724 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.066730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.066735 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.066788 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.067178 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.067184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.067189 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.067410 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:47.067494 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.068224 (+ 730us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.068395 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250109 21:03:47.069140 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.055030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:56003 (local address 127.12.126.125:40035)
0109 21:03:47.055245 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.055248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.056439 (+ 1191us) server_negotiation.cc:407] Connection header received
0109 21:03:47.056947 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.056950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.057007 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.057083 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:47.058839 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.059619 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.060506 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.060699 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.065354 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:47.065375 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:47.065378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:47.065413 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:47.067500 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.067974 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.067978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.067979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.068030 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.068281 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.068284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.068286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.068474 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:47.068538 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.068865 (+ 327us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.068993 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20250109 21:03:47.069147 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.055033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:39045 (local address 127.12.126.126:42099)
0109 21:03:47.055165 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.055169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.056368 (+ 1199us) server_negotiation.cc:407] Connection header received
0109 21:03:47.056915 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.056918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.056983 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.057085 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:47.058944 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.059762 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.060714 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.060942 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.065376 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:47.065401 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:47.065407 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:47.065445 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:47.067375 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.067908 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.067915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.067916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.067984 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.068287 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.068291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.068293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.068453 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:47.068542 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.068868 (+ 326us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.069011 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250109 21:03:47.070137 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.055255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:40035 (local address 127.12.126.65:56003)
0109 21:03:47.056348 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:47.056362 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:47.056725 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:47.057280 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:47.057283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:47.058402 (+ 1119us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:47.058665 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.058671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.059793 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.059796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:47.060374 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.060383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.060916 (+ 533us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.061951 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:47.061966 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:47.065164 (+ 3198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:47.067608 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:47.067610 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:47.067612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:47.067870 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:47.068132 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:47.068134 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:47.068136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:47.068176 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:47.068599 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:47.068601 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:47.068693 (+ 92us) client_negotiation.cc:769] Sending connection context
0109 21:03:47.069316 (+ 623us) client_negotiation.cc:240] Negotiation successful
0109 21:03:47.069549 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"mutex_wait_us":166,"spinlock_wait_cycles":62080,"thread_start_us":129,"threads_started":1}
I20250109 21:03:47.070158 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.055255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:37489 (local address 127.12.126.65:36207)
0109 21:03:47.055774 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:47.055815 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:47.056703 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:47.057257 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:47.057263 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:47.057895 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:47.058649 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.058665 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.059793 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.059796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:47.060374 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.060383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.060916 (+ 533us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.061606 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:47.061646 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:47.063801 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:47.065836 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:47.065844 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:47.065860 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:47.066605 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:47.066930 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:47.066932 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:47.066934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:47.067074 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:47.067578 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:47.067585 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:47.067909 (+ 324us) client_negotiation.cc:769] Sending connection context
0109 21:03:47.068862 (+ 953us) client_negotiation.cc:240] Negotiation successful
0109 21:03:47.069182 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"spinlock_wait_cycles":55936,"thread_start_us":123,"threads_started":1}
I20250109 21:03:47.070389 20441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.055317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.65:39045)
0109 21:03:47.056282 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:47.056298 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:47.056719 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:47.057255 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:47.057263 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:47.058414 (+ 1151us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:47.058768 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.058780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.059937 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.059943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:47.060572 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.060585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.061075 (+ 490us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.061796 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:47.061818 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:47.065215 (+ 3397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:47.067511 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:47.067514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:47.067516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:47.067791 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:47.068096 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:47.068102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:47.068105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:47.068163 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:47.068599 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:47.068601 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:47.068691 (+ 90us) client_negotiation.cc:769] Sending connection context
0109 21:03:47.069316 (+ 625us) client_negotiation.cc:240] Negotiation successful
0109 21:03:47.069465 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"spinlock_wait_cycles":17408,"thread_start_us":123,"threads_started":1}
I20250109 21:03:47.071507 20435 heartbeater.cc:346] Connected to a master server at 127.12.126.124:37489
I20250109 21:03:47.071560 20437 heartbeater.cc:346] Connected to a master server at 127.12.126.125:40035
I20250109 21:03:47.071560 20436 heartbeater.cc:346] Connected to a master server at 127.12.126.126:42099
I20250109 21:03:47.071770 20437 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:47.071771 20435 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:47.071770 20436 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:47.072436 20437 heartbeater.cc:510] Master 127.12.126.125:40035 requested a full tablet report, sending...
I20250109 21:03:47.072438 20435 heartbeater.cc:510] Master 127.12.126.124:37489 requested a full tablet report, sending...
I20250109 21:03:47.072485 20436 heartbeater.cc:510] Master 127.12.126.126:42099 requested a full tablet report, sending...
I20250109 21:03:47.073730 19446 ts_manager.cc:194] Re-registered known tserver with Master: ffe52c1a4d014d29a2a57c3120d158b1 (127.12.126.65:35925)
I20250109 21:03:47.073998 20049 ts_manager.cc:194] Registered new tserver with Master: ffe52c1a4d014d29a2a57c3120d158b1 (127.12.126.65:35925)
I20250109 21:03:47.073998 20231 ts_manager.cc:194] Registered new tserver with Master: ffe52c1a4d014d29a2a57c3120d158b1 (127.12.126.65:35925)
I20250109 21:03:47.074561 19446 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:39045
W20250109 21:03:47.192468 20442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:47.192721 20442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:47.192795 20442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:47.195858 20442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:47.195920 20442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:47.196012 20442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:03:47.200290 20442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.66.keytab
--principal=oryx/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:45203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=34881
--tserver_master_addrs=127.12.126.126:42099,127.12.126.125:40035,127.12.126.124:37489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:47.201313 20442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:47.202104 20442 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:47.210762 20448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:47.211340 20447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:47.212249 20450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:47.213006 20442 server_base.cc:1034] running on GCE node
I20250109 21:03:47.213348 20442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:47.213891 20442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:47.215034 20442 hybrid_clock.cc:648] HybridClock initialized: now 1736456627215018 us; error 38 us; skew 500 ppm
Jan 09 21:03:47 dist-test-slave-g0zb krb5kdc[19401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456627, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:47.217844 20442 init.cc:377] Logged in from keytab as oryx/127.12.126.66@KRBTEST.COM (short username oryx)
I20250109 21:03:47.218865 20442 webserver.cc:458] Webserver started at http://127.12.126.66:34881/ using document root <none> and password file <none>
I20250109 21:03:47.219424 20442 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:47.219488 20442 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:47.222982 20442 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20250109 21:03:47.224700 20457 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:47.225689 20442 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:47.225795 20442 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "44684bbb8586432aa2cc81943640a975"
format_stamp: "Formatted at 2025-01-09 21:03:43 on dist-test-slave-g0zb"
server_key: "c7bf889594351f2eaf95085de263bbb9"
server_key_iv: "96f5a2128bde31e39237ce33d7ff459f"
server_key_version: "encryptionkey@0"
I20250109 21:03:47.226195 20442 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:47.249987 20442 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:47.253086 20442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:47.253289 20442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:47.253870 20442 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:47.254665 20442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:47.254734 20442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:47.254793 20442 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:47.254850 20442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:47.263376 20442 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:45203
I20250109 21:03:47.263420 20570 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:45203 every 8 connection(s)
I20250109 21:03:47.264418 20442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:03:47.265983 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 20442
I20250109 21:03:47.266161 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 19776
I20250109 21:03:47.273831 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:36481
--local_ip_for_outbound_sockets=127.12.126.67
--tserver_master_addrs=127.12.126.126:42099
--webserver_port=43869
--webserver_interface=127.12.126.67
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.67.keytab
--principal=oryx/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.126.126:42099,127.12.126.125:40035
--tserver_master_addrs=127.12.126.126:42099,127.12.126.125:40035,127.12.126.124:37489 with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 09 21:03:47 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456627, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.124@KRBTEST.COM
Jan 09 21:03:47 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456627, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM
Jan 09 21:03:47 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456627, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.125@KRBTEST.COM
I20250109 21:03:47.281301 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.267791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:53805 (local address 127.12.126.124:37489)
0109 21:03:47.268059 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.268063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.268721 (+ 658us) server_negotiation.cc:407] Connection header received
0109 21:03:47.268941 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.268944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.269001 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.269054 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:47.271135 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.271635 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.272363 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.272547 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.276999 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:47.277023 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:47.277026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:47.277059 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:47.278982 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.279879 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.279882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.279884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.279927 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.280167 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.280170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.280172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.280344 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:47.280429 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.281007 (+ 578us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.281136 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20250109 21:03:47.281446 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.267927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:50575 (local address 127.12.126.125:40035)
0109 21:03:47.269560 (+ 1633us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.269567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.269583 (+ 16us) server_negotiation.cc:407] Connection header received
0109 21:03:47.269626 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.269632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.269687 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.269770 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:47.272163 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.272906 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.274075 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.274298 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.276867 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:47.276890 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:47.276897 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:47.276933 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:47.278832 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.279569 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.279573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.279575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.279626 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.279984 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.279990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.279993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.280212 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:47.280325 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.281042 (+ 717us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.281184 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1516}
I20250109 21:03:47.282481 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.266569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:40035 (local address 127.12.126.66:50575)
0109 21:03:47.267728 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:47.267774 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:47.269000 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:47.269891 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:47.269897 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:47.270142 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:47.270963 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.270983 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.273429 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.273431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:47.273941 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.273954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.274421 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.274973 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:47.274994 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:47.276721 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:47.278976 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:47.278985 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:47.278999 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:47.279471 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:47.279717 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:47.279722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:47.279726 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:47.279857 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:47.280405 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:47.280413 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:47.280740 (+ 327us) client_negotiation.cc:769] Sending connection context
0109 21:03:47.281586 (+ 846us) client_negotiation.cc:240] Negotiation successful
0109 21:03:47.281734 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"spinlock_wait_cycles":77184,"thread_start_us":142,"threads_started":1}
I20250109 21:03:47.282711 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.266569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.66:53859)
0109 21:03:47.268439 (+ 1870us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:47.268452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:47.268712 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:47.269357 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:47.269368 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:47.270124 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:47.270962 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.270981 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.272149 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.272152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:47.272679 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.272687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.272780 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.274244 (+ 1464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:47.274260 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:47.276492 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:47.279564 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:47.279566 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:47.279568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:47.279764 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:47.280155 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:47.280157 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:47.280159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:47.280199 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:47.281629 (+ 1430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:47.281634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:47.281719 (+ 85us) client_negotiation.cc:769] Sending connection context
0109 21:03:47.281807 (+ 88us) client_negotiation.cc:240] Negotiation successful
0109 21:03:47.281922 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1789,"mutex_wait_us":123,"spinlock_wait_cycles":14464,"thread_start_us":97,"threads_started":1}
I20250109 21:03:47.283142 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.266881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:53859 (local address 127.12.126.126:42099)
0109 21:03:47.267047 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.267053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.268534 (+ 1481us) server_negotiation.cc:407] Connection header received
0109 21:03:47.269075 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.269081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.269150 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.269255 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:47.271118 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.271949 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.273041 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.273271 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.276659 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:47.276685 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:47.276690 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:47.276725 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:47.279414 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.279864 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.279867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.279869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.279920 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.280486 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.280489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.280491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.280664 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:47.280755 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.282854 (+ 2099us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.282991 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250109 21:03:47.283001 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.266569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:37489 (local address 127.12.126.66:53805)
0109 21:03:47.268650 (+ 2081us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:47.268662 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:47.268780 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:47.269357 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:47.269368 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:47.270124 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:47.270962 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.270981 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.271756 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.271761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:47.272231 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.272239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.272774 (+ 535us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.273524 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:47.273564 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:47.276842 (+ 3278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:47.279108 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:47.279113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:47.279118 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:47.279478 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:47.280029 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:47.280032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:47.280034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:47.280074 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:47.280427 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:47.280430 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:47.280734 (+ 304us) client_negotiation.cc:769] Sending connection context
0109 21:03:47.280943 (+ 209us) client_negotiation.cc:240] Negotiation successful
0109 21:03:47.281223 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2005,"mutex_wait_us":1484,"spinlock_wait_cycles":9344,"thread_start_us":94,"threads_started":1}
I20250109 21:03:47.284216 20571 heartbeater.cc:346] Connected to a master server at 127.12.126.124:37489
I20250109 21:03:47.284257 20572 heartbeater.cc:346] Connected to a master server at 127.12.126.126:42099
I20250109 21:03:47.284296 20573 heartbeater.cc:346] Connected to a master server at 127.12.126.125:40035
I20250109 21:03:47.284443 20573 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:47.284443 20572 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:47.284443 20571 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:47.284938 20571 heartbeater.cc:510] Master 127.12.126.124:37489 requested a full tablet report, sending...
I20250109 21:03:47.284926 20572 heartbeater.cc:510] Master 127.12.126.126:42099 requested a full tablet report, sending...
I20250109 21:03:47.284924 20573 heartbeater.cc:510] Master 127.12.126.125:40035 requested a full tablet report, sending...
I20250109 21:03:47.286135 20231 ts_manager.cc:194] Registered new tserver with Master: 44684bbb8586432aa2cc81943640a975 (127.12.126.66:45203)
I20250109 21:03:47.286144 20049 ts_manager.cc:194] Registered new tserver with Master: 44684bbb8586432aa2cc81943640a975 (127.12.126.66:45203)
I20250109 21:03:47.286168 19446 ts_manager.cc:194] Re-registered known tserver with Master: 44684bbb8586432aa2cc81943640a975 (127.12.126.66:45203)
I20250109 21:03:47.286886 19446 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:53859
W20250109 21:03:47.385473 20578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:47.385746 20578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:47.385828 20578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:47.388916 20578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:47.388993 20578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:47.389106 20578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:03:47.393569 20578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.67.keytab
--principal=oryx/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:36481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=43869
--tserver_master_addrs=127.12.126.126:42099,127.12.126.125:40035,127.12.126.124:37489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:47.394721 20578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:47.395644 20578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:47.404496 20586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:47.405357 20583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:47.406069 20584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:47.407059 20578 server_base.cc:1034] running on GCE node
I20250109 21:03:47.407461 20578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:47.408047 20578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:47.409193 20578 hybrid_clock.cc:648] HybridClock initialized: now 1736456627409181 us; error 33 us; skew 500 ppm
Jan 09 21:03:47 dist-test-slave-g0zb krb5kdc[19401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456627, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:47.412179 20578 init.cc:377] Logged in from keytab as oryx/127.12.126.67@KRBTEST.COM (short username oryx)
I20250109 21:03:47.413065 20578 webserver.cc:458] Webserver started at http://127.12.126.67:43869/ using document root <none> and password file <none>
I20250109 21:03:47.413611 20578 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:47.413676 20578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:47.417133 20578 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:03:47.418903 20593 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:47.419936 20578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:47.420034 20578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "972591450de34e58b065db92abc49a77"
format_stamp: "Formatted at 2025-01-09 21:03:44 on dist-test-slave-g0zb"
server_key: "bba422824c9f1512547ae8739b51e64a"
server_key_iv: "57f3d5b726891e4981b5b146a83d07a8"
server_key_version: "encryptionkey@0"
I20250109 21:03:47.420465 20578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:47.462875 20578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:47.465962 20578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:47.466128 20578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:47.466686 20578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:47.467545 20578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:47.467593 20578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:47.467638 20578 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:47.467674 20578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:47.476262 20578 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:36481
I20250109 21:03:47.476305 20706 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:36481 every 8 connection(s)
I20250109 21:03:47.477331 20578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:03:47.480698 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 20578
Jan 09 21:03:47 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456627, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.125@KRBTEST.COM
Jan 09 21:03:47 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456627, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.124@KRBTEST.COM
Jan 09 21:03:47 dist-test-slave-g0zb krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456627, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM
I20250109 21:03:47.494532 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.479329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:40671 (local address 127.12.126.126:42099)
0109 21:03:47.479447 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.479452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.479977 (+ 525us) server_negotiation.cc:407] Connection header received
0109 21:03:47.481033 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.481036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.481100 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.481156 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:47.483431 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.484160 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.484904 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.485075 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.489546 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:47.489568 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:47.489574 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:47.489611 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:47.491421 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.492819 (+ 1398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.492823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.492825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.492880 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.493219 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.493223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.493234 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.493433 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:47.493537 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.494169 (+ 632us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.494312 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20250109 21:03:47.495536 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.479405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.67:40671)
0109 21:03:47.479848 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:47.479887 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:47.480788 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:47.481453 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:47.481465 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:47.482205 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:47.483025 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.483043 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.484281 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.484284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:47.484789 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.484797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.485176 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.486104 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:47.486142 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:47.489427 (+ 3285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:47.491541 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:47.491550 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:47.491567 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:47.491943 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:47.492970 (+ 1027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:47.492974 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:47.492976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:47.493104 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:47.493537 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:47.493543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:47.493865 (+ 322us) client_negotiation.cc:769] Sending connection context
0109 21:03:47.494101 (+ 236us) client_negotiation.cc:240] Negotiation successful
0109 21:03:47.494410 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"spinlock_wait_cycles":26368,"thread_start_us":105,"threads_started":1}
I20250109 21:03:47.496628 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.479435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:37489 (local address 127.12.126.67:50319)
0109 21:03:47.479848 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:47.479887 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:47.480794 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:47.481456 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:47.481465 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:47.482237 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:47.483026 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.483043 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.484082 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.484085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:47.484662 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.484674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.485165 (+ 491us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.486289 (+ 1124us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:47.486310 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:47.488770 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:47.495133 (+ 6363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:47.495136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:47.495138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:47.495362 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:47.495646 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:47.495649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:47.495653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:47.495713 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:47.496124 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:47.496127 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:47.496211 (+ 84us) client_negotiation.cc:769] Sending connection context
0109 21:03:47.496315 (+ 104us) client_negotiation.cc:240] Negotiation successful
0109 21:03:47.496456 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"spinlock_wait_cycles":122880,"thread_start_us":141,"threads_started":1}
I20250109 21:03:47.496675 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.479167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:50319 (local address 127.12.126.124:37489)
0109 21:03:47.479315 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.479319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.479979 (+ 660us) server_negotiation.cc:407] Connection header received
0109 21:03:47.481116 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.481119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.481176 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.481264 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:47.483195 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.483910 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.484790 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.484985 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.488945 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:47.488961 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:47.488964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:47.488992 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:47.494999 (+ 6007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.495488 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.495493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.495495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.495545 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.495808 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.495814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.495819 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.495989 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:47.496078 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.496395 (+ 317us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.496511 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250109 21:03:47.497368 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.483700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57484 (local address 127.12.126.124:37489)
0109 21:03:47.484658 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.484662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.485692 (+ 1030us) server_negotiation.cc:407] Connection header received
0109 21:03:47.485805 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.485808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.485854 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.485935 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:47.487686 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.488311 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.489978 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.490202 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.491447 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:47.491470 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:47.491476 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:47.491506 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:47.495138 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.495587 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.495592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.495597 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.495638 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.495882 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.495885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.495886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.495997 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:47.496974 (+ 977us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.497055 (+ 81us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.497172 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":56,"threads_started":1}
I20250109 21:03:47.497606 20708 heartbeater.cc:346] Connected to a master server at 127.12.126.126:42099
I20250109 21:03:47.497840 20708 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:47.497985 20707 heartbeater.cc:346] Connected to a master server at 127.12.126.124:37489
I20250109 21:03:47.498061 20707 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:47.498453 20708 heartbeater.cc:510] Master 127.12.126.126:42099 requested a full tablet report, sending...
I20250109 21:03:47.498452 20707 heartbeater.cc:510] Master 127.12.126.124:37489 requested a full tablet report, sending...
I20250109 21:03:47.499643 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.480263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:50765 (local address 127.12.126.125:40035)
0109 21:03:47.480421 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.480425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.480588 (+ 163us) server_negotiation.cc:407] Connection header received
0109 21:03:47.481380 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.481383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.481431 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.481512 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:47.483192 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.483918 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.484755 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.484943 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.488864 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:47.488887 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:47.488891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:47.488921 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:47.497342 (+ 8421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.497759 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.497763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.497764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.497823 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.498280 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.498284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.498285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.498459 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:47.498571 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.499367 (+ 796us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.499497 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20250109 21:03:47.499644 20231 ts_manager.cc:194] Registered new tserver with Master: 972591450de34e58b065db92abc49a77 (127.12.126.67:36481)
I20250109 21:03:47.499671 19446 ts_manager.cc:194] Re-registered known tserver with Master: 972591450de34e58b065db92abc49a77 (127.12.126.67:36481)
I20250109 21:03:47.500286 19446 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:40671
I20250109 21:03:47.500741 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.480244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:40035 (local address 127.12.126.67:50765)
0109 21:03:47.480482 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:47.480496 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:47.480838 (+ 342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:47.481631 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:47.481637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:47.482254 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:47.483025 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.483043 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.484060 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.484063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:47.484626 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.484635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.485165 (+ 530us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.486201 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:47.486220 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:47.488668 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:47.497506 (+ 8838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:47.497508 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:47.497509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:47.497667 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:47.498122 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:47.498125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:47.498127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:47.498182 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:47.499064 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:47.499069 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:47.499155 (+ 86us) client_negotiation.cc:769] Sending connection context
0109 21:03:47.499290 (+ 135us) client_negotiation.cc:240] Negotiation successful
0109 21:03:47.500622 (+ 1332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"spinlock_wait_cycles":6528,"thread_start_us":94,"threads_started":1}
I20250109 21:03:47.501374 20709 heartbeater.cc:346] Connected to a master server at 127.12.126.125:40035
I20250109 21:03:47.501433 20709 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:47.501555 20709 heartbeater.cc:510] Master 127.12.126.125:40035 requested a full tablet report, sending...
I20250109 21:03:47.502236 20049 ts_manager.cc:194] Registered new tserver with Master: 972591450de34e58b065db92abc49a77 (127.12.126.67:36481)
I20250109 21:03:47.502779 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.483299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51338 (local address 127.12.126.126:42099)
0109 21:03:47.485553 (+ 2254us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.485556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.485566 (+ 10us) server_negotiation.cc:407] Connection header received
0109 21:03:47.485593 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.485595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.485631 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.485729 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:47.488453 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.490147 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.491549 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.491775 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.493523 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:47.493539 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:47.493543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:47.493571 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:47.497324 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.499596 (+ 2272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.499599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.499601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.499646 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.501051 (+ 1405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.501055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.501056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.501230 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:47.501802 (+ 572us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.502513 (+ 711us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.502653 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2189,"thread_start_us":62,"threads_started":1}
I20250109 21:03:47.505633 19446 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 21:03:47.507856 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.482623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.12.126.125:40035)
0109 21:03:47.484199 (+ 1576us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.484203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.484688 (+ 485us) server_negotiation.cc:407] Connection header received
0109 21:03:47.484729 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.484734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.484790 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.484865 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:47.502265 (+ 17400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.502932 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.503567 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.503703 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.504898 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:47.504916 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:47.504919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:47.504946 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:47.506402 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.506771 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.506776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.506778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.506855 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.507062 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.507066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.507068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.507213 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:47.507346 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.507538 (+ 192us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.507660 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1505,"thread_start_us":81,"threads_started":1}
W20250109 21:03:47.508502 19446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:03:47.523751 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.519861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:45203 (local address 127.0.0.1:51128)
0109 21:03:47.520150 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:47.520166 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:47.520254 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:47.520831 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:47.520834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:47.520856 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:47.521111 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.521120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.522327 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.522330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:47.523309 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.523320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.523427 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.523450 (+ 23us) client_negotiation.cc:769] Sending connection context
0109 21:03:47.523518 (+ 68us) client_negotiation.cc:240] Negotiation successful
0109 21:03:47.523600 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":93,"threads_started":1}
I20250109 21:03:47.524585 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.520002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.12.126.66:45203)
0109 21:03:47.520306 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.520312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.520339 (+ 27us) server_negotiation.cc:407] Connection header received
0109 21:03:47.520449 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.520457 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.520691 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.520862 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:47.521230 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.522218 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.523447 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.524209 (+ 762us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.524311 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.524369 (+ 58us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.524459 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":101,"threads_started":1}
I20250109 21:03:47.524679 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.520423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:35925 (local address 127.0.0.1:48904)
0109 21:03:47.520656 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:47.520670 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:47.520748 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:47.522007 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:47.522010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:47.522030 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:47.522288 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.522300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.523600 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.523603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:47.524372 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.524385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.524469 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.524484 (+ 15us) client_negotiation.cc:769] Sending connection context
0109 21:03:47.524515 (+ 31us) client_negotiation.cc:240] Negotiation successful
0109 21:03:47.524558 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":100,"threads_started":1}
I20250109 21:03:47.525930 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.520597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.12.126.65:35925)
0109 21:03:47.521367 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.521373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.521396 (+ 23us) server_negotiation.cc:407] Connection header received
0109 21:03:47.521493 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.521498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.521849 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.522023 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:47.522407 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.523467 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.525246 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.525643 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.525729 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.525784 (+ 55us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.525838 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":68,"threads_started":1}
I20250109 21:03:47.526268 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.520711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:36481 (local address 127.0.0.1:57620)
0109 21:03:47.521892 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:47.521906 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:47.521995 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:47.522724 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:47.522726 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:47.522743 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:47.522965 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.522972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.524742 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.524745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:47.525934 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.525945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.526049 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.526064 (+ 15us) client_negotiation.cc:769] Sending connection context
0109 21:03:47.526107 (+ 43us) client_negotiation.cc:240] Negotiation successful
0109 21:03:47.526156 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1121,"thread_start_us":69,"threads_started":1}
I20250109 21:03:47.527597 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.521146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57620 (local address 127.12.126.67:36481)
0109 21:03:47.521626 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.521632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.522403 (+ 771us) server_negotiation.cc:407] Connection header received
0109 21:03:47.522480 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.522484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.522603 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.522727 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:47.523072 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.523985 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.526911 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.527287 (+ 376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.527366 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.527421 (+ 55us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.527491 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":101,"threads_started":1}
I20250109 21:03:47.527467 20505 tablet_service.cc:1467] Processing CreateTablet for tablet f31fe154d3f44fca9145d61372449323 (DEFAULT_TABLE table=test-table [id=5538c977042044fb8367ec653e5e843b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:47.528504 20505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31fe154d3f44fca9145d61372449323. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:47.528301 20369 tablet_service.cc:1467] Processing CreateTablet for tablet f31fe154d3f44fca9145d61372449323 (DEFAULT_TABLE table=test-table [id=5538c977042044fb8367ec653e5e843b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:47.529098 20369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31fe154d3f44fca9145d61372449323. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:47.535748 20741 tablet_bootstrap.cc:492] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1: Bootstrap starting.
I20250109 21:03:47.536706 20742 tablet_bootstrap.cc:492] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975: Bootstrap starting.
I20250109 21:03:47.537353 20641 tablet_service.cc:1467] Processing CreateTablet for tablet f31fe154d3f44fca9145d61372449323 (DEFAULT_TABLE table=test-table [id=5538c977042044fb8367ec653e5e843b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:47.538266 20641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31fe154d3f44fca9145d61372449323. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:47.538437 20742 tablet_bootstrap.cc:654] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:47.538503 20741 tablet_bootstrap.cc:654] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:47.539113 20742 log.cc:826] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:47.539240 20741 log.cc:826] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:47.540800 20741 tablet_bootstrap.cc:492] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1: No bootstrap required, opened a new log
I20250109 21:03:47.540874 20742 tablet_bootstrap.cc:492] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975: No bootstrap required, opened a new log
I20250109 21:03:47.540925 20741 ts_tablet_manager.cc:1397] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:03:47.541018 20742 ts_tablet_manager.cc:1397] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250109 21:03:47.543968 20741 raft_consensus.cc:357] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } } peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } }
I20250109 21:03:47.544127 20741 raft_consensus.cc:383] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:47.544157 20741 raft_consensus.cc:738] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe52c1a4d014d29a2a57c3120d158b1, State: Initialized, Role: FOLLOWER
I20250109 21:03:47.544143 20742 raft_consensus.cc:357] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } } peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } }
I20250109 21:03:47.544327 20742 raft_consensus.cc:383] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:47.544385 20742 raft_consensus.cc:738] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44684bbb8586432aa2cc81943640a975, State: Initialized, Role: FOLLOWER
I20250109 21:03:47.544548 20741 consensus_queue.cc:260] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } } peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } }
I20250109 21:03:47.544821 20742 consensus_queue.cc:260] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } } peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } }
I20250109 21:03:47.545580 20741 ts_tablet_manager.cc:1428] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:03:47.546154 20436 heartbeater.cc:502] Master 127.12.126.126:42099 was elected leader, sending a full tablet report...
I20250109 21:03:47.548353 20746 tablet_bootstrap.cc:492] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77: Bootstrap starting.
I20250109 21:03:47.548835 20572 heartbeater.cc:502] Master 127.12.126.126:42099 was elected leader, sending a full tablet report...
I20250109 21:03:47.551493 20742 ts_tablet_manager.cc:1428] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250109 21:03:47.552974 20746 tablet_bootstrap.cc:654] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:47.553884 20746 log.cc:826] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77: Log is configured to *not* fsync() on all Append() calls
W20250109 21:03:47.554762 20438 tablet.cc:2367] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:03:47.556999 20746 tablet_bootstrap.cc:492] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77: No bootstrap required, opened a new log
I20250109 21:03:47.557196 20746 ts_tablet_manager.cc:1397] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20250109 21:03:47.560248 20746 raft_consensus.cc:357] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } } peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } }
I20250109 21:03:47.560422 20746 raft_consensus.cc:383] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:47.560513 20746 raft_consensus.cc:738] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 972591450de34e58b065db92abc49a77, State: Initialized, Role: FOLLOWER
I20250109 21:03:47.560920 20746 consensus_queue.cc:260] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } } peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } }
I20250109 21:03:47.561544 20708 heartbeater.cc:502] Master 127.12.126.126:42099 was elected leader, sending a full tablet report...
I20250109 21:03:47.561713 20746 ts_tablet_manager.cc:1428] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20250109 21:03:47.728595 20710 tablet.cc:2367] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:03:47.740324 20745 raft_consensus.cc:491] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:03:47.740550 20745 raft_consensus.cc:513] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } } peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } }
I20250109 21:03:47.741698 20745 leader_election.cc:290] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44684bbb8586432aa2cc81943640a975 (127.12.126.66:45203), 972591450de34e58b065db92abc49a77 (127.12.126.67:36481)
I20250109 21:03:47.744828 20751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.741955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:36481 (local address 127.12.126.65:37753)
0109 21:03:47.742397 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:47.742413 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:47.742536 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:47.742779 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:47.742782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:47.742824 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:47.743096 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.743108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.743823 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.743826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:47.744403 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.744411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.744537 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.744555 (+ 18us) client_negotiation.cc:769] Sending connection context
0109 21:03:47.744606 (+ 51us) client_negotiation.cc:240] Negotiation successful
0109 21:03:47.744664 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":50,"thread_start_us":74,"threads_started":1}
I20250109 21:03:47.744890 20750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.741951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:45203 (local address 127.12.126.65:40287)
0109 21:03:47.742330 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:47.742341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:47.742442 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:47.742704 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:47.742709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:47.742729 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:47.743076 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.743086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.743810 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.743815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:47.744394 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:47.744402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.744502 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.744529 (+ 27us) client_negotiation.cc:769] Sending connection context
0109 21:03:47.744582 (+ 53us) client_negotiation.cc:240] Negotiation successful
0109 21:03:47.744648 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":91,"threads_started":1}
I20250109 21:03:47.745260 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.742075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:37753 (local address 127.12.126.67:36481)
0109 21:03:47.742229 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.742234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.742529 (+ 295us) server_negotiation.cc:407] Connection header received
0109 21:03:47.742630 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.742632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.742684 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.742762 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:47.743228 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.743713 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.744544 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.744998 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.745044 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.745095 (+ 51us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.745155 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250109 21:03:47.745296 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.742033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:40287 (local address 127.12.126.66:45203)
0109 21:03:47.742208 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.742214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.742425 (+ 211us) server_negotiation.cc:407] Connection header received
0109 21:03:47.742547 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.742550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.742611 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.742681 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:47.743168 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.743700 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.744511 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.744977 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.745025 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.745078 (+ 53us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.745145 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20250109 21:03:47.745942 20525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31fe154d3f44fca9145d61372449323" candidate_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44684bbb8586432aa2cc81943640a975" is_pre_election: true
I20250109 21:03:47.746174 20661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31fe154d3f44fca9145d61372449323" candidate_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "972591450de34e58b065db92abc49a77" is_pre_election: true
I20250109 21:03:47.746240 20525 raft_consensus.cc:2463] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffe52c1a4d014d29a2a57c3120d158b1 in term 0.
I20250109 21:03:47.746420 20661 raft_consensus.cc:2463] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffe52c1a4d014d29a2a57c3120d158b1 in term 0.
I20250109 21:03:47.746721 20323 leader_election.cc:304] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44684bbb8586432aa2cc81943640a975, ffe52c1a4d014d29a2a57c3120d158b1; no voters:
I20250109 21:03:47.746992 20745 raft_consensus.cc:2798] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:03:47.747058 20745 raft_consensus.cc:491] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:03:47.747093 20745 raft_consensus.cc:3054] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:47.748055 20745 raft_consensus.cc:513] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } } peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } }
I20250109 21:03:47.748411 20745 leader_election.cc:290] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [CANDIDATE]: Term 1 election: Requested vote from peers 44684bbb8586432aa2cc81943640a975 (127.12.126.66:45203), 972591450de34e58b065db92abc49a77 (127.12.126.67:36481)
I20250109 21:03:47.748772 20525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31fe154d3f44fca9145d61372449323" candidate_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44684bbb8586432aa2cc81943640a975"
I20250109 21:03:47.748782 20661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31fe154d3f44fca9145d61372449323" candidate_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "972591450de34e58b065db92abc49a77"
I20250109 21:03:47.748875 20525 raft_consensus.cc:3054] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:47.748886 20661 raft_consensus.cc:3054] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:47.749810 20661 raft_consensus.cc:2463] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe52c1a4d014d29a2a57c3120d158b1 in term 1.
I20250109 21:03:47.750138 20322 leader_election.cc:304] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 972591450de34e58b065db92abc49a77, ffe52c1a4d014d29a2a57c3120d158b1; no voters:
I20250109 21:03:47.750219 20525 raft_consensus.cc:2463] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe52c1a4d014d29a2a57c3120d158b1 in term 1.
I20250109 21:03:47.750344 20745 raft_consensus.cc:2798] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:47.750527 20745 raft_consensus.cc:695] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [term 1 LEADER]: Becoming Leader. State: Replica: ffe52c1a4d014d29a2a57c3120d158b1, State: Running, Role: LEADER
I20250109 21:03:47.751094 20745 consensus_queue.cc:237] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } } peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } }
I20250109 21:03:47.754338 19446 catalog_manager.cc:5526] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffe52c1a4d014d29a2a57c3120d158b1 (127.12.126.65). New cstate: current_term: 1 leader_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } health_report { overall_health: UNKNOWN } } }
W20250109 21:03:47.765877 20574 tablet.cc:2367] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:03:47.818569 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.815742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48910 (local address 127.12.126.65:35925)
0109 21:03:47.815880 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.815884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.815897 (+ 13us) server_negotiation.cc:407] Connection header received
0109 21:03:47.815983 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.815985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.816039 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.816138 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:47.816405 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.816896 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.817491 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.817632 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.817724 (+ 92us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:47.818177 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:47.818297 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.818329 (+ 32us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.818402 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250109 21:03:47.824270 20661 raft_consensus.cc:1270] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77 [term 1 FOLLOWER]: Refusing update from remote peer ffe52c1a4d014d29a2a57c3120d158b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:47.824270 20525 raft_consensus.cc:1270] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975 [term 1 FOLLOWER]: Refusing update from remote peer ffe52c1a4d014d29a2a57c3120d158b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:47.824934 20745 consensus_queue.cc:1035] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:47.825120 20752 consensus_queue.cc:1035] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:47.833964 20757 mvcc.cc:204] Tried to move back new op lower bound from 7112526347560927232 to 7112526347270742016. Current Snapshot: MvccSnapshot[applied={T|T < 7112526347560927232}]
I20250109 21:03:47.843081 19445 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51338:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:47.843240 19445 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51338:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:47.847371 19445 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 77fb9c5a4bed4855be740888c1397e6c: Sending DeleteTablet for 3 replicas of tablet f31fe154d3f44fca9145d61372449323
I20250109 21:03:47.848011 20505 tablet_service.cc:1514] Processing DeleteTablet for tablet f31fe154d3f44fca9145d61372449323 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:47 UTC) from {username='oryx', principal='oryx/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:51128
I20250109 21:03:47.848250 20369 tablet_service.cc:1514] Processing DeleteTablet for tablet f31fe154d3f44fca9145d61372449323 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:47 UTC) from {username='oryx', principal='oryx/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:48904
I20250109 21:03:47.848369 20641 tablet_service.cc:1514] Processing DeleteTablet for tablet f31fe154d3f44fca9145d61372449323 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:47 UTC) from {username='oryx', principal='oryx/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:57620
I20250109 21:03:47.848728 20765 tablet_replica.cc:331] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1: stopping tablet replica
I20250109 21:03:47.849504 20765 raft_consensus.cc:2238] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:03:47.850128 20765 raft_consensus.cc:2267] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:47.851483 20765 ts_tablet_manager.cc:1905] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:47.853524 20766 tablet_replica.cc:331] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77: stopping tablet replica
I20250109 21:03:47.853572 20765 ts_tablet_manager.cc:1918] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:03:47.853832 20765 log.cc:1198] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-0/wal/wals/f31fe154d3f44fca9145d61372449323
I20250109 21:03:47.853832 20766 raft_consensus.cc:2238] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:47.854192 20766 raft_consensus.cc:2267] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:47.854627 20765 ts_tablet_manager.cc:1939] T f31fe154d3f44fca9145d61372449323 P ffe52c1a4d014d29a2a57c3120d158b1: Deleting consensus metadata
I20250109 21:03:47.855222 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 19414
I20250109 21:03:47.860970 20766 ts_tablet_manager.cc:1905] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:47.861382 20764 tablet_replica.cc:331] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975: stopping tablet replica
I20250109 21:03:47.861676 20764 raft_consensus.cc:2238] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:47.862006 20764 raft_consensus.cc:2267] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:47.863461 20764 ts_tablet_manager.cc:1905] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 21:03:47.863744 20325 connection.cc:537] server connection from 127.0.0.1:48904 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250109 21:03:47.864563 20766 ts_tablet_manager.cc:1918] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:03:47.864748 20766 log.cc:1198] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/f31fe154d3f44fca9145d61372449323
I20250109 21:03:47.865206 20766 ts_tablet_manager.cc:1939] T f31fe154d3f44fca9145d61372449323 P 972591450de34e58b065db92abc49a77: Deleting consensus metadata
I20250109 21:03:47.866354 20764 ts_tablet_manager.cc:1918] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:03:47.866466 20764 log.cc:1198] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/f31fe154d3f44fca9145d61372449323
W20250109 21:03:47.866603 20595 connection.cc:441] server connection from 127.0.0.1:57620 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57620 (request call id 1) could send its response
I20250109 21:03:47.866781 20764 ts_tablet_manager.cc:1939] T f31fe154d3f44fca9145d61372449323 P 44684bbb8586432aa2cc81943640a975: Deleting consensus metadata
W20250109 21:03:47.867982 20458 connection.cc:441] server connection from 127.0.0.1:51128 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51128 (request call id 1) could send its response
I20250109 21:03:47.876631 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.867880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53368 (local address 127.12.126.125:40035)
0109 21:03:47.868121 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.868126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.868481 (+ 355us) server_negotiation.cc:407] Connection header received
0109 21:03:47.868680 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.868684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.868736 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.868787 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:47.869931 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.870730 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.871619 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.871811 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.873142 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:47.873162 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:47.873166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:47.873197 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:47.874968 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.875398 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.875402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.875404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.875458 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.875670 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.875673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.875676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.875882 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:47.876002 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.876223 (+ 221us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.876482 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20250109 21:03:47.877242 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:47.867965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57498 (local address 127.12.126.124:37489)
0109 21:03:47.868224 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:47.868229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:47.868604 (+ 375us) server_negotiation.cc:407] Connection header received
0109 21:03:47.868731 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:47.868734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:47.868791 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:47.868870 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:47.870012 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.870523 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:47.871294 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:47.871505 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:47.873482 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:47.873500 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:47.873504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:47.873530 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:47.875240 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.875743 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.875747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.875749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.875801 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:47.876056 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:47.876061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:47.876063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:47.876258 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:47.876355 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:47.876968 (+ 613us) server_negotiation.cc:299] Negotiation successful
0109 21:03:47.877099 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20250109 21:03:48.838049 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:48.837544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.66:52617)
0109 21:03:48.837831 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:48.837909 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":93,"threads_started":1}
W20250109 21:03:48.838536 20572 heartbeater.cc:643] Failed to heartbeat to 127.12.126.126:42099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
I20250109 21:03:48.839897 20775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:48.839404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.67:37975)
0109 21:03:48.839719 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:48.839787 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":73,"threads_started":1}
W20250109 21:03:48.840165 20708 heartbeater.cc:643] Failed to heartbeat to 127.12.126.126:42099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
I20250109 21:03:48.840162 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:48.839759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.65:41995)
0109 21:03:48.839981 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:48.840068 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":72,"threads_started":1}
W20250109 21:03:48.840404 20436 heartbeater.cc:643] Failed to heartbeat to 127.12.126.126:42099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
I20250109 21:03:49.347632 20777 raft_consensus.cc:491] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77fb9c5a4bed4855be740888c1397e6c)
I20250109 21:03:49.347826 20777 raft_consensus.cc:513] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: false } }
I20250109 21:03:49.348899 20777 leader_election.cc:290] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77fb9c5a4bed4855be740888c1397e6c (127.12.126.126:42099), fed07e4fff2e449a95dd07aa8cf7925f (127.12.126.125:40035)
I20250109 21:03:49.349503 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.349078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.0.0.1:51820)
0109 21:03:49.349328 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.349413 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":61,"threads_started":1}
I20250109 21:03:49.349428 20059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" is_pre_election: true
I20250109 21:03:49.349678 20059 raft_consensus.cc:2463] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a956bf10bf8d4c388a73b2a1dc6a1cf6 in term 1.
I20250109 21:03:49.349797 20779 raft_consensus.cc:491] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77fb9c5a4bed4855be740888c1397e6c)
W20250109 21:03:49.349820 20224 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
I20250109 21:03:49.349885 20779 raft_consensus.cc:513] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: false } }
I20250109 21:03:49.350078 20226 leader_election.cc:304] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a956bf10bf8d4c388a73b2a1dc6a1cf6, fed07e4fff2e449a95dd07aa8cf7925f; no voters:
I20250109 21:03:49.350322 20777 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250109 21:03:49.350399 20777 raft_consensus.cc:491] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77fb9c5a4bed4855be740888c1397e6c)
I20250109 21:03:49.350441 20777 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:03:49.350675 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.350444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.0.0.1:51822)
0109 21:03:49.350567 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.350622 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20250109 21:03:49.350777 20224 leader_election.cc:336] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 77fb9c5a4bed4855be740888c1397e6c (127.12.126.126:42099): Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
I20250109 21:03:49.351181 20779 leader_election.cc:290] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77fb9c5a4bed4855be740888c1397e6c (127.12.126.126:42099), a956bf10bf8d4c388a73b2a1dc6a1cf6 (127.12.126.124:37489)
I20250109 21:03:49.351425 20241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" is_pre_election: true
I20250109 21:03:49.351549 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.351162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.0.0.1:51834)
0109 21:03:49.351420 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.351474 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":70,"threads_started":1}
W20250109 21:03:49.351688 20034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
I20250109 21:03:49.352226 20777 raft_consensus.cc:513] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: false } }
I20250109 21:03:49.352499 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.352264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.0.0.1:51836)
0109 21:03:49.352378 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.352432 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20250109 21:03:49.352505 20241 raft_consensus.cc:2388] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fed07e4fff2e449a95dd07aa8cf7925f in current term 2: Already voted for candidate a956bf10bf8d4c388a73b2a1dc6a1cf6 in this term.
I20250109 21:03:49.352639 20777 leader_election.cc:290] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [CANDIDATE]: Term 2 election: Requested vote from peers 77fb9c5a4bed4855be740888c1397e6c (127.12.126.126:42099), fed07e4fff2e449a95dd07aa8cf7925f (127.12.126.125:40035)
W20250109 21:03:49.352931 20034 leader_election.cc:336] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 77fb9c5a4bed4855be740888c1397e6c (127.12.126.126:42099): Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
I20250109 21:03:49.352972 20059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fed07e4fff2e449a95dd07aa8cf7925f"
I20250109 21:03:49.353070 20059 raft_consensus.cc:3054] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:03:49.353080 20033 leader_election.cc:304] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fed07e4fff2e449a95dd07aa8cf7925f; no voters: 77fb9c5a4bed4855be740888c1397e6c, a956bf10bf8d4c388a73b2a1dc6a1cf6
I20250109 21:03:49.354061 20059 raft_consensus.cc:2463] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a956bf10bf8d4c388a73b2a1dc6a1cf6 in term 2.
I20250109 21:03:49.354166 20779 raft_consensus.cc:2743] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 21:03:49.354369 20226 leader_election.cc:304] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a956bf10bf8d4c388a73b2a1dc6a1cf6, fed07e4fff2e449a95dd07aa8cf7925f; no voters:
I20250109 21:03:49.354454 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.352751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.0.0.1:51844)
0109 21:03:49.352844 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.352893 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20250109 21:03:49.354661 20777 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 2 FOLLOWER]: Leader election won for term 2
I20250109 21:03:49.354887 20777 raft_consensus.cc:695] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [term 2 LEADER]: Becoming Leader. State: Replica: a956bf10bf8d4c388a73b2a1dc6a1cf6, State: Running, Role: LEADER
I20250109 21:03:49.355124 20777 consensus_queue.cc:237] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: false } }
I20250109 21:03:49.355209 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.354947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.0.0.1:51860)
0109 21:03:49.355081 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.355123 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20250109 21:03:49.355300 20224 leader_election.cc:336] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 77fb9c5a4bed4855be740888c1397e6c (127.12.126.126:42099): Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
I20250109 21:03:49.356182 20783 sys_catalog.cc:455] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a956bf10bf8d4c388a73b2a1dc6a1cf6. Latest consensus state: current_term: 2 leader_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: false } } }
I20250109 21:03:49.356317 20783 sys_catalog.cc:458] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:49.356590 20785 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:03:49.358681 20785 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:03:49.359083 20785 catalog_manager.cc:1260] Loaded cluster ID: 4bd8a54b43d446b0b0d221053a3d2adc
I20250109 21:03:49.359130 20785 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:03:49.359835 20785 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:03:49.360286 20785 catalog_manager.cc:5910] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: Loaded TSK: 0
I20250109 21:03:49.360975 20785 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:03:49.385129 20233 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:03:49.387786 20233 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:03:49.390656 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.390404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.0.0.1:51866)
0109 21:03:49.390543 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.390590 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20250109 21:03:49.390758 20059 raft_consensus.cc:1270] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [term 2 FOLLOWER]: Refusing update from remote peer a956bf10bf8d4c388a73b2a1dc6a1cf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250109 21:03:49.391275 20783 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250109 21:03:49.391358 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.391146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.0.0.1:51874)
0109 21:03:49.391263 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.391305 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20250109 21:03:49.391577 20224 consensus_peers.cc:487] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 -> Peer 77fb9c5a4bed4855be740888c1397e6c (127.12.126.126:42099): Couldn't send request to peer 77fb9c5a4bed4855be740888c1397e6c. Status: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 21:03:49.392757 20779 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a956bf10bf8d4c388a73b2a1dc6a1cf6. Latest consensus state: current_term: 2 leader_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: false } } }
I20250109 21:03:49.392889 20779 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:03:49.394061 20777 sys_catalog.cc:455] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: false } } }
I20250109 21:03:49.394177 20777 sys_catalog.cc:458] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:49.394665 20779 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "77fb9c5a4bed4855be740888c1397e6c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42099 } } peers { permanent_uuid: "fed07e4fff2e449a95dd07aa8cf7925f" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 40035 } attrs { promote: false } } peers { permanent_uuid: "a956bf10bf8d4c388a73b2a1dc6a1cf6" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 37489 } attrs { promote: false } } }
I20250109 21:03:49.394766 20779 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed07e4fff2e449a95dd07aa8cf7925f [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:03:49.403254 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.400035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:35925 (local address 127.0.0.1:48918)
0109 21:03:49.400155 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.400168 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:49.400255 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:49.400848 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:49.400852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:49.400871 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:49.401135 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:49.401146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.402240 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:49.402244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:49.402875 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:49.402888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.402995 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:49.403015 (+ 20us) client_negotiation.cc:769] Sending connection context
0109 21:03:49.403067 (+ 52us) client_negotiation.cc:240] Negotiation successful
0109 21:03:49.403127 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20250109 21:03:49.404032 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.400575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:36481 (local address 127.0.0.1:57630)
0109 21:03:49.401293 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.401301 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:49.401370 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:49.401606 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:49.401609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:49.401624 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:49.401822 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:49.401827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.402768 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:49.402771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:49.403678 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:49.403688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.403780 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:49.403795 (+ 15us) client_negotiation.cc:769] Sending connection context
0109 21:03:49.403837 (+ 42us) client_negotiation.cc:240] Negotiation successful
0109 21:03:49.403885 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":240,"thread_start_us":67,"threads_started":1}
I20250109 21:03:49.404582 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.400329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48918 (local address 127.12.126.65:35925)
0109 21:03:49.400583 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:49.400589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:49.400603 (+ 14us) server_negotiation.cc:407] Connection header received
0109 21:03:49.400671 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:49.400673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:49.400726 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:49.401242 (+ 516us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:49.401354 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.402140 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:49.403906 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.404356 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:49.404391 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:49.404435 (+ 44us) server_negotiation.cc:299] Negotiation successful
0109 21:03:49.404486 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
I20250109 21:03:49.404963 20795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.400801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51140 (local address 127.12.126.66:45203)
0109 21:03:49.401466 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:49.401471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:49.401483 (+ 12us) server_negotiation.cc:407] Connection header received
0109 21:03:49.401528 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:49.401531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:49.401574 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:49.401649 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:49.402129 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.402881 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:49.404227 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.404692 (+ 465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:49.404728 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:49.404779 (+ 51us) server_negotiation.cc:299] Negotiation successful
0109 21:03:49.404838 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":68,"threads_started":1}
I20250109 21:03:49.404963 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.400708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57630 (local address 127.12.126.67:36481)
0109 21:03:49.401091 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:49.401097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:49.401365 (+ 268us) server_negotiation.cc:407] Connection header received
0109 21:03:49.401474 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:49.401477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:49.401520 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:49.401592 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:49.401923 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.402643 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:49.404204 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.404692 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:49.404728 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:49.404779 (+ 51us) server_negotiation.cc:299] Negotiation successful
0109 21:03:49.404838 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":53,"threads_started":1}
I20250109 21:03:49.405079 20369 tablet_service.cc:1467] Processing CreateTablet for tablet dbde02ffd67a47c8beda81b0853428b6 (DEFAULT_TABLE table=test-table [id=0bd68a17324e4a538c722ce8ae7132eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:49.405373 20369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbde02ffd67a47c8beda81b0853428b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:49.405488 20641 tablet_service.cc:1467] Processing CreateTablet for tablet dbde02ffd67a47c8beda81b0853428b6 (DEFAULT_TABLE table=test-table [id=0bd68a17324e4a538c722ce8ae7132eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:49.405735 20641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbde02ffd67a47c8beda81b0853428b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:49.407047 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.400575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:45203 (local address 127.0.0.1:51140)
0109 21:03:49.401062 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.401075 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:49.401153 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:49.401762 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:49.401765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:49.401782 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:49.402011 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:49.402017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.402989 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:49.402992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:49.403889 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:49.403896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.403977 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:49.403990 (+ 13us) client_negotiation.cc:769] Sending connection context
0109 21:03:49.404034 (+ 44us) client_negotiation.cc:240] Negotiation successful
0109 21:03:49.404087 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":103,"threads_started":1}
I20250109 21:03:49.407527 20797 tablet_bootstrap.cc:492] T dbde02ffd67a47c8beda81b0853428b6 P ffe52c1a4d014d29a2a57c3120d158b1: Bootstrap starting.
I20250109 21:03:49.407639 20798 tablet_bootstrap.cc:492] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77: Bootstrap starting.
I20250109 21:03:49.408514 20798 tablet_bootstrap.cc:654] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:49.408566 20797 tablet_bootstrap.cc:654] T dbde02ffd67a47c8beda81b0853428b6 P ffe52c1a4d014d29a2a57c3120d158b1: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:49.408548 20505 tablet_service.cc:1467] Processing CreateTablet for tablet dbde02ffd67a47c8beda81b0853428b6 (DEFAULT_TABLE table=test-table [id=0bd68a17324e4a538c722ce8ae7132eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:49.408807 20505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbde02ffd67a47c8beda81b0853428b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:49.409679 20797 tablet_bootstrap.cc:492] T dbde02ffd67a47c8beda81b0853428b6 P ffe52c1a4d014d29a2a57c3120d158b1: No bootstrap required, opened a new log
I20250109 21:03:49.409757 20797 ts_tablet_manager.cc:1397] T dbde02ffd67a47c8beda81b0853428b6 P ffe52c1a4d014d29a2a57c3120d158b1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:49.409780 20798 tablet_bootstrap.cc:492] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77: No bootstrap required, opened a new log
I20250109 21:03:49.409860 20798 ts_tablet_manager.cc:1397] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250109 21:03:49.410079 20797 raft_consensus.cc:357] T dbde02ffd67a47c8beda81b0853428b6 P ffe52c1a4d014d29a2a57c3120d158b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } } peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } }
I20250109 21:03:49.410157 20797 raft_consensus.cc:383] T dbde02ffd67a47c8beda81b0853428b6 P ffe52c1a4d014d29a2a57c3120d158b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:49.410176 20797 raft_consensus.cc:738] T dbde02ffd67a47c8beda81b0853428b6 P ffe52c1a4d014d29a2a57c3120d158b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe52c1a4d014d29a2a57c3120d158b1, State: Initialized, Role: FOLLOWER
I20250109 21:03:49.410290 20797 consensus_queue.cc:260] T dbde02ffd67a47c8beda81b0853428b6 P ffe52c1a4d014d29a2a57c3120d158b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } } peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } }
I20250109 21:03:49.410284 20798 raft_consensus.cc:357] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } } peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } }
I20250109 21:03:49.410408 20798 raft_consensus.cc:383] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:49.410439 20798 raft_consensus.cc:738] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 972591450de34e58b065db92abc49a77, State: Initialized, Role: FOLLOWER
I20250109 21:03:49.410589 20797 ts_tablet_manager.cc:1428] T dbde02ffd67a47c8beda81b0853428b6 P ffe52c1a4d014d29a2a57c3120d158b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 21:03:49.410584 20798 consensus_queue.cc:260] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } } peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } }
I20250109 21:03:49.411412 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.411035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.65:49955)
0109 21:03:49.411270 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.411337 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":84,"threads_started":1}
I20250109 21:03:49.412947 20798 ts_tablet_manager.cc:1428] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250109 21:03:49.413257 20802 tablet_bootstrap.cc:492] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975: Bootstrap starting.
I20250109 21:03:49.414299 20802 tablet_bootstrap.cc:654] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:49.414894 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.414510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.67:37067)
0109 21:03:49.414742 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.414787 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
I20250109 21:03:49.414992 20435 heartbeater.cc:502] Master 127.12.126.124:37489 was elected leader, sending a full tablet report...
I20250109 21:03:49.415635 20802 tablet_bootstrap.cc:492] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975: No bootstrap required, opened a new log
I20250109 21:03:49.416034 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.415356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.65:36863)
0109 21:03:49.415841 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.415947 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20250109 21:03:49.416064 20802 ts_tablet_manager.cc:1397] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20250109 21:03:49.414978 20707 heartbeater.cc:502] Master 127.12.126.124:37489 was elected leader, sending a full tablet report...
I20250109 21:03:49.416519 20802 raft_consensus.cc:357] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } } peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } }
I20250109 21:03:49.416638 20802 raft_consensus.cc:383] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:49.416683 20802 raft_consensus.cc:738] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44684bbb8586432aa2cc81943640a975, State: Initialized, Role: FOLLOWER
I20250109 21:03:49.416842 20802 consensus_queue.cc:260] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } } peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } }
I20250109 21:03:49.417227 20802 ts_tablet_manager.cc:1428] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250109 21:03:49.417945 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.417714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.67:33657)
0109 21:03:49.417820 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.417876 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20250109 21:03:49.418175 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.417715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.66:44519)
0109 21:03:49.417961 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.418092 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":102,"threads_started":1}
I20250109 21:03:49.419312 20571 heartbeater.cc:502] Master 127.12.126.124:37489 was elected leader, sending a full tablet report...
I20250109 21:03:49.419756 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.419577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.66:46269)
0109 21:03:49.419662 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.419703 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20250109 21:03:49.569067 20803 raft_consensus.cc:491] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:03:49.569245 20803 raft_consensus.cc:513] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } } peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } }
I20250109 21:03:49.570425 20803 leader_election.cc:290] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe52c1a4d014d29a2a57c3120d158b1 (127.12.126.65:35925), 44684bbb8586432aa2cc81943640a975 (127.12.126.66:45203)
I20250109 21:03:49.573961 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.570633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:45203 (local address 127.12.126.67:58963)
0109 21:03:49.571172 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.571185 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:49.571280 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:49.571572 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:49.571575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:49.571588 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:49.571845 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:49.571852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.572731 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:49.572734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:49.573508 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:49.573515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.573626 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:49.573646 (+ 20us) client_negotiation.cc:769] Sending connection context
0109 21:03:49.573701 (+ 55us) client_negotiation.cc:240] Negotiation successful
0109 21:03:49.573778 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":235,"thread_start_us":125,"threads_started":1}
I20250109 21:03:49.573961 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.570633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:35925 (local address 127.12.126.67:38639)
0109 21:03:49.570874 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.570918 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:49.571084 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:49.571371 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:49.571375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:49.571395 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:49.571651 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:49.571662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.572596 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:49.572599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:49.573466 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:49.573487 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.573622 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:49.573646 (+ 24us) client_negotiation.cc:769] Sending connection context
0109 21:03:49.573701 (+ 55us) client_negotiation.cc:240] Negotiation successful
0109 21:03:49.573766 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20250109 21:03:49.574390 20795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.570764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:58963 (local address 127.12.126.66:45203)
0109 21:03:49.570978 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:49.570985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:49.571245 (+ 260us) server_negotiation.cc:407] Connection header received
0109 21:03:49.571415 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:49.571420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:49.571474 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:49.571536 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:49.571950 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.572642 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:49.573612 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.574119 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:49.574160 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:49.574214 (+ 54us) server_negotiation.cc:299] Negotiation successful
0109 21:03:49.574268 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20250109 21:03:49.574390 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.570788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:38639 (local address 127.12.126.65:35925)
0109 21:03:49.570955 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:49.570960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:49.571083 (+ 123us) server_negotiation.cc:407] Connection header received
0109 21:03:49.571215 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:49.571217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:49.571263 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:49.571337 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:49.571779 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.572483 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:49.573616 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.574119 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:49.574160 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:49.574214 (+ 54us) server_negotiation.cc:299] Negotiation successful
0109 21:03:49.574270 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250109 21:03:49.574865 20525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbde02ffd67a47c8beda81b0853428b6" candidate_uuid: "972591450de34e58b065db92abc49a77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44684bbb8586432aa2cc81943640a975" is_pre_election: true
I20250109 21:03:49.574926 20389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbde02ffd67a47c8beda81b0853428b6" candidate_uuid: "972591450de34e58b065db92abc49a77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" is_pre_election: true
I20250109 21:03:49.575003 20525 raft_consensus.cc:2463] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 972591450de34e58b065db92abc49a77 in term 0.
I20250109 21:03:49.575148 20389 raft_consensus.cc:2463] T dbde02ffd67a47c8beda81b0853428b6 P ffe52c1a4d014d29a2a57c3120d158b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 972591450de34e58b065db92abc49a77 in term 0.
I20250109 21:03:49.575403 20595 leader_election.cc:304] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44684bbb8586432aa2cc81943640a975, 972591450de34e58b065db92abc49a77; no voters:
I20250109 21:03:49.575683 20803 raft_consensus.cc:2798] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:03:49.575745 20803 raft_consensus.cc:491] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:03:49.575804 20803 raft_consensus.cc:3054] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:49.576757 20803 raft_consensus.cc:513] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } } peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } }
I20250109 21:03:49.577091 20803 leader_election.cc:290] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [CANDIDATE]: Term 1 election: Requested vote from peers ffe52c1a4d014d29a2a57c3120d158b1 (127.12.126.65:35925), 44684bbb8586432aa2cc81943640a975 (127.12.126.66:45203)
I20250109 21:03:49.577502 20525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbde02ffd67a47c8beda81b0853428b6" candidate_uuid: "972591450de34e58b065db92abc49a77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44684bbb8586432aa2cc81943640a975"
I20250109 21:03:49.577497 20389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbde02ffd67a47c8beda81b0853428b6" candidate_uuid: "972591450de34e58b065db92abc49a77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe52c1a4d014d29a2a57c3120d158b1"
I20250109 21:03:49.577615 20525 raft_consensus.cc:3054] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:49.577622 20389 raft_consensus.cc:3054] T dbde02ffd67a47c8beda81b0853428b6 P ffe52c1a4d014d29a2a57c3120d158b1 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:49.578435 20525 raft_consensus.cc:2463] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 972591450de34e58b065db92abc49a77 in term 1.
I20250109 21:03:49.578559 20389 raft_consensus.cc:2463] T dbde02ffd67a47c8beda81b0853428b6 P ffe52c1a4d014d29a2a57c3120d158b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 972591450de34e58b065db92abc49a77 in term 1.
I20250109 21:03:49.578723 20595 leader_election.cc:304] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44684bbb8586432aa2cc81943640a975, 972591450de34e58b065db92abc49a77; no voters:
I20250109 21:03:49.578922 20803 raft_consensus.cc:2798] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:49.579098 20803 raft_consensus.cc:695] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [term 1 LEADER]: Becoming Leader. State: Replica: 972591450de34e58b065db92abc49a77, State: Running, Role: LEADER
I20250109 21:03:49.579349 20803 consensus_queue.cc:237] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } } peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } }
I20250109 21:03:49.579802 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.579574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.67:47611)
0109 21:03:49.579677 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.579731 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20250109 21:03:49.582151 20237 catalog_manager.cc:5526] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 972591450de34e58b065db92abc49a77 (127.12.126.67). New cstate: current_term: 1 leader_uuid: "972591450de34e58b065db92abc49a77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "972591450de34e58b065db92abc49a77" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 36481 } health_report { overall_health: HEALTHY } } }
I20250109 21:03:49.595835 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.593075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57640 (local address 127.12.126.67:36481)
0109 21:03:49.593206 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:49.593210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:49.593221 (+ 11us) server_negotiation.cc:407] Connection header received
0109 21:03:49.593308 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:49.593310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:49.593365 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:49.593462 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:49.593739 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.594212 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:49.594911 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:49.595043 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:49.595101 (+ 58us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:49.595550 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:49.595641 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:49.595685 (+ 44us) server_negotiation.cc:299] Negotiation successful
0109 21:03:49.595728 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250109 21:03:49.598573 20525 raft_consensus.cc:1270] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975 [term 1 FOLLOWER]: Refusing update from remote peer 972591450de34e58b065db92abc49a77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:49.598742 20389 raft_consensus.cc:1270] T dbde02ffd67a47c8beda81b0853428b6 P ffe52c1a4d014d29a2a57c3120d158b1 [term 1 FOLLOWER]: Refusing update from remote peer 972591450de34e58b065db92abc49a77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:49.599092 20803 consensus_queue.cc:1035] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44684bbb8586432aa2cc81943640a975" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:49.599318 20810 consensus_queue.cc:1035] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe52c1a4d014d29a2a57c3120d158b1" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:49.602402 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.602080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.65:37625)
0109 21:03:49.602238 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.602311 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20250109 21:03:49.602421 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.601565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.66:41483)
0109 21:03:49.602285 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.602344 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":629}
I20250109 21:03:49.604281 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.604007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.67:35769)
0109 21:03:49.604136 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.604188 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20250109 21:03:49.605949 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.605672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.67:43057)
0109 21:03:49.605823 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.605872 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20250109 21:03:49.605947 20814 mvcc.cc:204] Tried to move back new op lower bound from 7112526354831003648 to 7112526354759491584. Current Snapshot: MvccSnapshot[applied={T|T < 7112526354831003648}]
I20250109 21:03:49.606202 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.605947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.66:39721)
0109 21:03:49.606090 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.606140 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20250109 21:03:49.608191 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.607865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42099 (local address 127.12.126.65:60553)
0109 21:03:49.608034 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.608096 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20250109 21:03:49.612854 20237 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57498:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:49.613018 20237 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57498:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:49.616971 20237 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a956bf10bf8d4c388a73b2a1dc6a1cf6: Sending DeleteTablet for 3 replicas of tablet dbde02ffd67a47c8beda81b0853428b6
I20250109 21:03:49.617751 20369 tablet_service.cc:1514] Processing DeleteTablet for tablet dbde02ffd67a47c8beda81b0853428b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:49 UTC) from {username='oryx', principal='oryx/127.12.126.124@KRBTEST.COM'} at 127.0.0.1:48918
I20250109 21:03:49.617765 20641 tablet_service.cc:1514] Processing DeleteTablet for tablet dbde02ffd67a47c8beda81b0853428b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:49 UTC) from {username='oryx', principal='oryx/127.12.126.124@KRBTEST.COM'} at 127.0.0.1:57630
I20250109 21:03:49.618167 20823 tablet_replica.cc:331] T dbde02ffd67a47c8beda81b0853428b6 P ffe52c1a4d014d29a2a57c3120d158b1: stopping tablet replica
I20250109 21:03:49.618296 20823 raft_consensus.cc:2238] T dbde02ffd67a47c8beda81b0853428b6 P ffe52c1a4d014d29a2a57c3120d158b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:49.618402 20505 tablet_service.cc:1514] Processing DeleteTablet for tablet dbde02ffd67a47c8beda81b0853428b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:49 UTC) from {username='oryx', principal='oryx/127.12.126.124@KRBTEST.COM'} at 127.0.0.1:51140
I20250109 21:03:49.618443 20823 raft_consensus.cc:2267] T dbde02ffd67a47c8beda81b0853428b6 P ffe52c1a4d014d29a2a57c3120d158b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:49.618675 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 20306
I20250109 21:03:49.619969 20824 tablet_replica.cc:331] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975: stopping tablet replica
I20250109 21:03:49.620184 20824 raft_consensus.cc:2238] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:49.620407 20824 raft_consensus.cc:2267] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:49.620762 20822 tablet_replica.cc:331] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77: stopping tablet replica
I20250109 21:03:49.620958 20822 raft_consensus.cc:2238] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:03:49.621317 20822 raft_consensus.cc:2267] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:49.621313 20824 ts_tablet_manager.cc:1905] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:49.621999 20822 ts_tablet_manager.cc:1905] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:49.623406 20824 ts_tablet_manager.cc:1918] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:03:49.623481 20824 log.cc:1198] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/dbde02ffd67a47c8beda81b0853428b6
I20250109 21:03:49.623782 20824 ts_tablet_manager.cc:1939] T dbde02ffd67a47c8beda81b0853428b6 P 44684bbb8586432aa2cc81943640a975: Deleting consensus metadata
I20250109 21:03:49.624517 20224 catalog_manager.cc:4872] TS 44684bbb8586432aa2cc81943640a975 (127.12.126.66:45203): tablet dbde02ffd67a47c8beda81b0853428b6 (table test-table [id=0bd68a17324e4a538c722ce8ae7132eb]) successfully deleted
I20250109 21:03:49.625383 20822 ts_tablet_manager.cc:1918] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:03:49.625468 20822 log.cc:1198] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/dbde02ffd67a47c8beda81b0853428b6
I20250109 21:03:49.625790 20822 ts_tablet_manager.cc:1939] T dbde02ffd67a47c8beda81b0853428b6 P 972591450de34e58b065db92abc49a77: Deleting consensus metadata
I20250109 21:03:49.626588 20223 catalog_manager.cc:4872] TS 972591450de34e58b065db92abc49a77 (127.12.126.67:36481): tablet dbde02ffd67a47c8beda81b0853428b6 (table test-table [id=0bd68a17324e4a538c722ce8ae7132eb]) successfully deleted
I20250109 21:03:49.628415 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 20442
I20250109 21:03:49.629387 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:49.628643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:35925 (local address 127.0.0.1:48930)
0109 21:03:49.629201 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:49.629279 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.65:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441}
W20250109 21:03:49.630935 20223 catalog_manager.cc:4615] TS ffe52c1a4d014d29a2a57c3120d158b1 (127.12.126.65:35925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbde02ffd67a47c8beda81b0853428b6: Network error: Client connection negotiation failed: client connection to 127.12.126.65:35925: connect: Connection refused (error 111)
I20250109 21:03:49.635645 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 20578
I20250109 21:03:49.641952 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 19910
I20250109 21:03:49.648011 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 20021
2025-01-09T21:03:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7453 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:03:49 dist-test-slave-g0zb krb5kdc[20827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:03:49 dist-test-slave-g0zb krb5kdc[20827](info): set up 2 sockets
Jan 09 21:03:49 dist-test-slave-g0zb krb5kdc[20827](info): commencing operation
krb5kdc: starting...
W20250109 21:03:50.652218 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:03:50 dist-test-slave-g0zb krb5kdc[20827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:03:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:03:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.126.keytab.
Entry for principal oryx/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.126.keytab.
I20250109 21:03:50.767951 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:35621
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.126.keytab
--principal=oryx/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:32777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:35621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:50.872970 20840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:50.873250 20840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:50.873335 20840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:50.876770 20840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:50.876850 20840 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250109 21:03:50.876881 20840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:50.876921 20840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:50.876961 20840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:50.877000 20840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:03:50.881500 20840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:32777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:35621
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.126.keytab
--principal=oryx/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:35621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:50.882773 20840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:50.883787 20840 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:50.891319 20848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:50.891950 20845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:50.892809 20846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:50.893652 20840 server_base.cc:1034] running on GCE node
I20250109 21:03:50.894145 20840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:50.895107 20840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:50.896270 20840 hybrid_clock.cc:648] HybridClock initialized: now 1736456630896248 us; error 36 us; skew 500 ppm
Jan 09 21:03:50 dist-test-slave-g0zb krb5kdc[20827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456630, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:50.898924 20840 init.cc:377] Logged in from keytab as oryx/127.12.126.126@KRBTEST.COM (short username oryx)
I20250109 21:03:50.899876 20840 webserver.cc:458] Webserver started at http://127.12.126.126:35485/ using document root <none> and password file <none>
I20250109 21:03:50.900434 20840 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:50.900497 20840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:50.900677 20840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:50.902568 20840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "ece6e8158f834a93afd6176c9d06b967"
format_stamp: "Formatted at 2025-01-09 21:03:50 on dist-test-slave-g0zb"
server_key: "950e02ab45ad45cae00126a703e296b1"
server_key_iv: "3a5a98cc1201815fce90d13f1398e1e1"
server_key_version: "encryptionkey@0"
I20250109 21:03:50.903116 20840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "ece6e8158f834a93afd6176c9d06b967"
format_stamp: "Formatted at 2025-01-09 21:03:50 on dist-test-slave-g0zb"
server_key: "950e02ab45ad45cae00126a703e296b1"
server_key_iv: "3a5a98cc1201815fce90d13f1398e1e1"
server_key_version: "encryptionkey@0"
I20250109 21:03:50.906574 20840 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250109 21:03:50.908913 20855 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:50.909987 20840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250109 21:03:50.910106 20840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "ece6e8158f834a93afd6176c9d06b967"
format_stamp: "Formatted at 2025-01-09 21:03:50 on dist-test-slave-g0zb"
server_key: "950e02ab45ad45cae00126a703e296b1"
server_key_iv: "3a5a98cc1201815fce90d13f1398e1e1"
server_key_version: "encryptionkey@0"
I20250109 21:03:50.910203 20840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:50.946235 20840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:50.949193 20840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:50.949388 20840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:50.957999 20917 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:35621 every 8 connection(s)
I20250109 21:03:50.958005 20840 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:35621
I20250109 21:03:50.959087 20840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:03:50.961572 20918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:50.964208 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 20840
I20250109 21:03:50.964309 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:03:50.964538 12793 external_mini_cluster.cc:1453] Setting key bf2428816f876fe0ca2b0c8d29c8bc9b
I20250109 21:03:50.966905 20918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967: Bootstrap starting.
I20250109 21:03:50.969381 20918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:50.970223 20918 log.cc:826] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:50.972316 20918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967: No bootstrap required, opened a new log
Jan 09 21:03:50 dist-test-slave-g0zb krb5kdc[20827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM
I20250109 21:03:50.975845 20918 raft_consensus.cc:357] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece6e8158f834a93afd6176c9d06b967" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35621 } }
I20250109 21:03:50.976037 20918 raft_consensus.cc:383] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:50.976078 20918 raft_consensus.cc:738] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece6e8158f834a93afd6176c9d06b967, State: Initialized, Role: FOLLOWER
I20250109 21:03:50.976583 20918 consensus_queue.cc:260] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece6e8158f834a93afd6176c9d06b967" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35621 } }
I20250109 21:03:50.976732 20918 raft_consensus.cc:397] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:03:50.976800 20918 raft_consensus.cc:491] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:03:50.976919 20918 raft_consensus.cc:3054] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:50.977427 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:50.965787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.12.126.126:35621)
0109 21:03:50.966186 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:50.966195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:50.966248 (+ 53us) server_negotiation.cc:407] Connection header received
0109 21:03:50.966791 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:50.966847 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:50.967171 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:50.967504 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:50.968584 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:50.969385 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:50.970249 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:50.970538 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:50.973058 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:50.973077 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:50.973088 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:50.973121 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:50.974932 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:50.975385 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:50.975389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:50.975393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:50.975463 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:50.975686 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:50.975688 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:50.975689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:50.976010 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:50.976222 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:50.976467 (+ 245us) server_negotiation.cc:299] Negotiation successful
0109 21:03:50.976728 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
I20250109 21:03:50.977818 20918 raft_consensus.cc:513] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece6e8158f834a93afd6176c9d06b967" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35621 } }
I20250109 21:03:50.978133 20918 leader_election.cc:304] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ece6e8158f834a93afd6176c9d06b967; no voters:
I20250109 21:03:50.978420 20918 leader_election.cc:290] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:03:50.978518 20923 raft_consensus.cc:2798] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:50.978731 20923 raft_consensus.cc:695] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967 [term 1 LEADER]: Becoming Leader. State: Replica: ece6e8158f834a93afd6176c9d06b967, State: Running, Role: LEADER
I20250109 21:03:50.979038 20923 consensus_queue.cc:237] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece6e8158f834a93afd6176c9d06b967" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35621 } }
I20250109 21:03:50.979631 20918 sys_catalog.cc:564] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:50.980710 20925 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ece6e8158f834a93afd6176c9d06b967. Latest consensus state: current_term: 1 leader_uuid: "ece6e8158f834a93afd6176c9d06b967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece6e8158f834a93afd6176c9d06b967" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35621 } } }
I20250109 21:03:50.980823 20925 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:50.981029 20924 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ece6e8158f834a93afd6176c9d06b967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece6e8158f834a93afd6176c9d06b967" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35621 } } }
I20250109 21:03:50.981105 20924 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967 [sys.catalog]: This master's current role is: LEADER
W20250109 21:03:50.984067 20938 catalog_manager.cc:1559] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:03:50.984136 20938 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 21:03:50.984207 20932 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:03:50.984853 20932 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:03:50.989769 20932 catalog_manager.cc:1348] Generated new cluster ID: 3114fd0fab8648698ec7162953ff62f2
I20250109 21:03:50.989841 20932 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:03:51.005815 20932 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:03:51.006743 20932 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:03:51.012410 20932 catalog_manager.cc:5899] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967: Generated new TSK 0
I20250109 21:03:51.013067 20932 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.65.keytab.
Entry for principal oryx/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.65.keytab.
I20250109 21:03:51.050539 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35621
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.65.keytab
--principal=oryx/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:32777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:51.164927 20944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:51.165196 20944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:51.165297 20944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:51.168506 20944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:51.168584 20944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:51.168684 20944 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250109 21:03:51.168759 20944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:03:51.173210 20944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:32777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.65.keytab
--principal=oryx/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.126.126:35621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:51.174352 20944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:51.175247 20944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:51.184063 20949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:51.184913 20950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:51.185617 20952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:51.186584 20944 server_base.cc:1034] running on GCE node
I20250109 21:03:51.187058 20944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:51.187744 20944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:51.188923 20944 hybrid_clock.cc:648] HybridClock initialized: now 1736456631188911 us; error 26 us; skew 500 ppm
Jan 09 21:03:51 dist-test-slave-g0zb krb5kdc[20827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456631, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:51.191751 20944 init.cc:377] Logged in from keytab as oryx/127.12.126.65@KRBTEST.COM (short username oryx)
I20250109 21:03:51.192773 20944 webserver.cc:458] Webserver started at http://127.12.126.65:36847/ using document root <none> and password file <none>
I20250109 21:03:51.193367 20944 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:51.193429 20944 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:51.193606 20944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:51.195495 20944 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "4976a64c48044865ab6c8917922e8f92"
format_stamp: "Formatted at 2025-01-09 21:03:51 on dist-test-slave-g0zb"
server_key: "d1f1fd918cd884b7a47fd55662c7c90f"
server_key_iv: "ca310fa12867fda7601909d58b5e1a4b"
server_key_version: "encryptionkey@0"
I20250109 21:03:51.195976 20944 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "4976a64c48044865ab6c8917922e8f92"
format_stamp: "Formatted at 2025-01-09 21:03:51 on dist-test-slave-g0zb"
server_key: "d1f1fd918cd884b7a47fd55662c7c90f"
server_key_iv: "ca310fa12867fda7601909d58b5e1a4b"
server_key_version: "encryptionkey@0"
I20250109 21:03:51.199502 20944 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:03:51.201879 20959 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:51.203019 20944 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:03:51.203126 20944 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "4976a64c48044865ab6c8917922e8f92"
format_stamp: "Formatted at 2025-01-09 21:03:51 on dist-test-slave-g0zb"
server_key: "d1f1fd918cd884b7a47fd55662c7c90f"
server_key_iv: "ca310fa12867fda7601909d58b5e1a4b"
server_key_version: "encryptionkey@0"
I20250109 21:03:51.203235 20944 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:51.220831 20944 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:51.223680 20944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:51.223857 20944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:51.225221 20944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:51.225273 20944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:51.225333 20944 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:51.225363 20944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 09 21:03:51 dist-test-slave-g0zb krb5kdc[20827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456631, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM
I20250109 21:03:51.235630 20944 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:43869
I20250109 21:03:51.236338 21074 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:43869 every 8 connection(s)
I20250109 21:03:51.236738 20944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:03:51.236950 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 20944
I20250109 21:03:51.237080 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:03:51.237393 12793 external_mini_cluster.cc:1453] Setting key fbdbd7bba6f2ae9d8e55ff7c48ede325
I20250109 21:03:51.238351 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.225969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:46087 (local address 127.12.126.126:35621)
0109 21:03:51.226096 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:51.226100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:51.227030 (+ 930us) server_negotiation.cc:407] Connection header received
0109 21:03:51.227993 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:51.227997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:51.228068 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:51.228135 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:51.230028 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.230850 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.231555 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.231806 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.234508 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:51.234537 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:51.234540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:51.234568 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:51.236143 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:51.236681 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:51.236685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:51.236687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:51.236747 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:51.237103 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:51.237109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:51.237112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:51.237329 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:51.237428 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:51.238044 (+ 616us) server_negotiation.cc:299] Negotiation successful
0109 21:03:51.238183 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250109 21:03:51.239039 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.226388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35621 (local address 127.12.126.65:46087)
0109 21:03:51.226893 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:51.226929 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:51.227723 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:51.228585 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:51.228599 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:51.229069 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:51.229868 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.229888 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.230963 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.230968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:51.231422 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.231430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.231693 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.232416 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:51.232477 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:51.234347 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:51.236261 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:51.236266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:51.236278 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:51.236582 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:51.236860 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:51.236864 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:51.236866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:51.236987 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:51.237491 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:51.237497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:51.237778 (+ 281us) client_negotiation.cc:769] Sending connection context
0109 21:03:51.238000 (+ 222us) client_negotiation.cc:240] Negotiation successful
0109 21:03:51.238283 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":141,"threads_started":1}
I20250109 21:03:51.240770 21075 heartbeater.cc:346] Connected to a master server at 127.12.126.126:35621
I20250109 21:03:51.240932 21075 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:51.241463 21075 heartbeater.cc:510] Master 127.12.126.126:35621 requested a full tablet report, sending...
I20250109 21:03:51.243012 20871 ts_manager.cc:194] Registered new tserver with Master: 4976a64c48044865ab6c8917922e8f92 (127.12.126.65:43869)
I20250109 21:03:51.244688 20871 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:46087
WARNING: no policy specified for oryx/127.12.126.66@KRBTEST.COM; defaulting to no policy
I20250109 21:03:51.248922 21082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.242289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35621 (local address 127.12.126.65:44475)
0109 21:03:51.242582 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:51.242594 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:51.242688 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:51.242934 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:51.242937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:51.243121 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:51.243369 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.243377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.244271 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.244274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:51.244642 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.244649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.244757 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.245617 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:51.245631 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:51.245924 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:51.247659 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:51.247663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:51.247665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:51.247839 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:51.248052 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:51.248055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:51.248059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:51.248102 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:51.248404 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:51.248407 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:51.248485 (+ 78us) client_negotiation.cc:769] Sending connection context
0109 21:03:51.248578 (+ 93us) client_negotiation.cc:240] Negotiation successful
0109 21:03:51.248719 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":112,"threads_started":1}
I20250109 21:03:51.248953 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.242328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:44475 (local address 127.12.126.126:35621)
0109 21:03:51.242460 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:51.242465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:51.242658 (+ 193us) server_negotiation.cc:407] Connection header received
0109 21:03:51.242778 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:51.242781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:51.242850 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:51.242922 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:51.243478 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.244141 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.244747 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.244950 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.246041 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:51.246059 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:51.246063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:51.246090 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:51.247551 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:51.247926 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:51.247932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:51.247934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:51.247975 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:51.248178 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:51.248181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:51.248183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:51.248320 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:51.248407 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:51.248637 (+ 230us) server_negotiation.cc:299] Negotiation successful
0109 21:03:51.248746 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.126.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.66.keytab.
Entry for principal oryx/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.66.keytab.
I20250109 21:03:51.265936 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35621
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.66.keytab
--principal=oryx/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:32777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:51.374715 21084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:51.375021 21084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:51.375115 21084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:51.378346 21084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:51.378427 21084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:51.378484 21084 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250109 21:03:51.378561 21084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:03:51.383322 21084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:32777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.66.keytab
--principal=oryx/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.126.126:35621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:51.384529 21084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:51.385387 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:51.394338 21092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:51.394968 21090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:51.395925 21089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:51.396708 21084 server_base.cc:1034] running on GCE node
I20250109 21:03:51.397104 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:51.397728 21084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:51.398864 21084 hybrid_clock.cc:648] HybridClock initialized: now 1736456631398849 us; error 29 us; skew 500 ppm
Jan 09 21:03:51 dist-test-slave-g0zb krb5kdc[20827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456631, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:51.401575 21084 init.cc:377] Logged in from keytab as oryx/127.12.126.66@KRBTEST.COM (short username oryx)
I20250109 21:03:51.402524 21084 webserver.cc:458] Webserver started at http://127.12.126.66:39835/ using document root <none> and password file <none>
I20250109 21:03:51.403119 21084 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:51.403180 21084 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:51.403388 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:51.405238 21084 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "4eec2002b5d9418c84bc049d10f686ff"
format_stamp: "Formatted at 2025-01-09 21:03:51 on dist-test-slave-g0zb"
server_key: "a097b1032b210c57f7906b42c9b124c1"
server_key_iv: "68ab7490b33e229db3fbe7efcdaecdf2"
server_key_version: "encryptionkey@0"
I20250109 21:03:51.405711 21084 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "4eec2002b5d9418c84bc049d10f686ff"
format_stamp: "Formatted at 2025-01-09 21:03:51 on dist-test-slave-g0zb"
server_key: "a097b1032b210c57f7906b42c9b124c1"
server_key_iv: "68ab7490b33e229db3fbe7efcdaecdf2"
server_key_version: "encryptionkey@0"
I20250109 21:03:51.409174 21084 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20250109 21:03:51.411271 21099 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:51.412246 21084 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:51.412364 21084 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "4eec2002b5d9418c84bc049d10f686ff"
format_stamp: "Formatted at 2025-01-09 21:03:51 on dist-test-slave-g0zb"
server_key: "a097b1032b210c57f7906b42c9b124c1"
server_key_iv: "68ab7490b33e229db3fbe7efcdaecdf2"
server_key_version: "encryptionkey@0"
I20250109 21:03:51.412448 21084 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:51.431006 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:51.433765 21084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:51.433948 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:51.435479 21084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:51.435536 21084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:51.435595 21084 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:51.435626 21084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 09 21:03:51 dist-test-slave-g0zb krb5kdc[20827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456631, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM
I20250109 21:03:51.445750 21084 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:33389
I20250109 21:03:51.446146 21214 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:33389 every 8 connection(s)
I20250109 21:03:51.446894 21084 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:03:51.448766 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.436094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:54693 (local address 127.12.126.126:35621)
0109 21:03:51.436238 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:51.436243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:51.436999 (+ 756us) server_negotiation.cc:407] Connection header received
0109 21:03:51.437979 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:51.437984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:51.438051 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:51.438115 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:51.439895 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.440463 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.441169 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.441362 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.444898 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:51.444922 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:51.444925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:51.444955 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:51.446608 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:51.447163 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:51.447167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:51.447169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:51.447217 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:51.447509 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:51.447513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:51.447516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:51.447710 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:51.447802 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:51.448455 (+ 653us) server_negotiation.cc:299] Negotiation successful
0109 21:03:51.448589 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250109 21:03:51.449338 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.436411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35621 (local address 127.12.126.66:54693)
0109 21:03:51.436844 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:51.436886 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:51.437727 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:51.438565 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:51.438578 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:51.439078 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:51.439739 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.439753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.440582 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.440588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:51.441036 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.441044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.441313 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.442202 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:51.442243 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:51.444721 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:51.446727 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:51.446733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:51.446744 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:51.447066 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:51.447317 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:51.447320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:51.447322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:51.447418 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:51.447988 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:51.447992 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:51.448214 (+ 222us) client_negotiation.cc:769] Sending connection context
0109 21:03:51.448389 (+ 175us) client_negotiation.cc:240] Negotiation successful
0109 21:03:51.448692 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":112,"threads_started":1}
I20250109 21:03:51.450464 21215 heartbeater.cc:346] Connected to a master server at 127.12.126.126:35621
I20250109 21:03:51.450604 21215 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:51.451153 21215 heartbeater.cc:510] Master 127.12.126.126:35621 requested a full tablet report, sending...
I20250109 21:03:51.451903 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 21084
I20250109 21:03:51.452031 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:03:51.452311 12793 external_mini_cluster.cc:1453] Setting key 8abd9b29010b267dddba4168e39b0eeb
I20250109 21:03:51.452309 20872 ts_manager.cc:194] Registered new tserver with Master: 4eec2002b5d9418c84bc049d10f686ff (127.12.126.66:33389)
I20250109 21:03:51.453047 20872 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:54693
I20250109 21:03:51.460104 21221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.452798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35621 (local address 127.12.126.66:36687)
0109 21:03:51.453087 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:51.453102 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:51.453216 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:51.453725 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:51.453728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:51.453917 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:51.454187 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.454208 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.455242 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.455247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:51.455848 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.455862 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.455978 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.456499 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:51.456514 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:51.456785 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:51.458847 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:51.458849 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:51.458851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:51.459014 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:51.459257 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:51.459259 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:51.459260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:51.459295 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:51.459700 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:51.459702 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:51.459772 (+ 70us) client_negotiation.cc:769] Sending connection context
0109 21:03:51.459853 (+ 81us) client_negotiation.cc:240] Negotiation successful
0109 21:03:51.459957 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":111,"threads_started":1}
I20250109 21:03:51.460413 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.452862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:36687 (local address 127.12.126.126:35621)
0109 21:03:51.453525 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:51.453528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:51.453538 (+ 10us) server_negotiation.cc:407] Connection header received
0109 21:03:51.453568 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:51.453571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:51.453616 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:51.453694 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:51.454339 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.455095 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.455974 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.456227 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.456905 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:51.456927 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:51.456933 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:51.456964 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:51.458698 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:51.459106 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:51.459111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:51.459115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:51.459174 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:51.459365 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:51.459368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:51.459369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:51.459611 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:51.459718 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:51.459930 (+ 212us) server_negotiation.cc:299] Negotiation successful
0109 21:03:51.460053 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599}
WARNING: no policy specified for oryx/127.12.126.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.126.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.67.keytab.
Entry for principal oryx/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.67.keytab.
I20250109 21:03:51.479322 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35621
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.67.keytab
--principal=oryx/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:32777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:51.587169 21224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:51.587430 21224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:51.587540 21224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:51.590911 21224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:51.590987 21224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:51.591068 21224 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250109 21:03:51.591140 21224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:03:51.595629 21224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:32777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.67.keytab
--principal=oryx/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.126.126:35621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:51.596787 21224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:51.597698 21224 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:51.606455 21230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:51.607379 21229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:51.607998 21232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:51.609045 21224 server_base.cc:1034] running on GCE node
I20250109 21:03:51.609459 21224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:51.610026 21224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:51.611183 21224 hybrid_clock.cc:648] HybridClock initialized: now 1736456631611167 us; error 29 us; skew 500 ppm
Jan 09 21:03:51 dist-test-slave-g0zb krb5kdc[20827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456631, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:51.613929 21224 init.cc:377] Logged in from keytab as oryx/127.12.126.67@KRBTEST.COM (short username oryx)
I20250109 21:03:51.615047 21224 webserver.cc:458] Webserver started at http://127.12.126.67:45975/ using document root <none> and password file <none>
I20250109 21:03:51.615649 21224 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:51.615706 21224 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:51.615900 21224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:51.617794 21224 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "cb51b5220e604548a6da753363f95292"
format_stamp: "Formatted at 2025-01-09 21:03:51 on dist-test-slave-g0zb"
server_key: "6c02c5c7aab7f11c41b0ed4dd9cb8655"
server_key_iv: "60d53d788ffb81e5f250c75ff83f5eb5"
server_key_version: "encryptionkey@0"
I20250109 21:03:51.618283 21224 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "cb51b5220e604548a6da753363f95292"
format_stamp: "Formatted at 2025-01-09 21:03:51 on dist-test-slave-g0zb"
server_key: "6c02c5c7aab7f11c41b0ed4dd9cb8655"
server_key_iv: "60d53d788ffb81e5f250c75ff83f5eb5"
server_key_version: "encryptionkey@0"
I20250109 21:03:51.621905 21224 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250109 21:03:51.624259 21239 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:51.625322 21224 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:03:51.625447 21224 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "cb51b5220e604548a6da753363f95292"
format_stamp: "Formatted at 2025-01-09 21:03:51 on dist-test-slave-g0zb"
server_key: "6c02c5c7aab7f11c41b0ed4dd9cb8655"
server_key_iv: "60d53d788ffb81e5f250c75ff83f5eb5"
server_key_version: "encryptionkey@0"
I20250109 21:03:51.625560 21224 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:51.640964 21224 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:51.644183 21224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:51.644335 21224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:51.645782 21224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:51.645838 21224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:51.645879 21224 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:51.645918 21224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 09 21:03:51 dist-test-slave-g0zb krb5kdc[20827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456631, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM
I20250109 21:03:51.656446 21224 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:45411
I20250109 21:03:51.656497 21354 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:45411 every 8 connection(s)
I20250109 21:03:51.657375 21224 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:03:51.658885 21248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.646865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35621 (local address 127.12.126.67:47745)
0109 21:03:51.647312 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:51.647348 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:51.648089 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:51.648762 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:51.648775 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:51.649227 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:51.649865 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.649879 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.650854 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.650857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:51.651284 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.651296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.651518 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.652230 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:51.652262 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:51.654562 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:51.656549 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:51.656556 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:51.656570 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:51.656824 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:51.657050 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:51.657053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:51.657055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:51.657134 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:51.657513 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:51.657518 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:51.657756 (+ 238us) client_negotiation.cc:769] Sending connection context
0109 21:03:51.657968 (+ 212us) client_negotiation.cc:240] Negotiation successful
0109 21:03:51.658230 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":124,"threads_started":1}
I20250109 21:03:51.659206 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.646416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:47745 (local address 127.12.126.126:35621)
0109 21:03:51.646540 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:51.646544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:51.647448 (+ 904us) server_negotiation.cc:407] Connection header received
0109 21:03:51.648321 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:51.648325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:51.648392 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:51.648475 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:51.650059 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.650715 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.651441 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.651688 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.654751 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:51.654776 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:51.654779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:51.654836 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:51.656423 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:51.656926 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:51.656929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:51.656930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:51.656975 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:51.657212 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:51.657215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:51.657217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:51.657370 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:51.657458 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:51.658934 (+ 1476us) server_negotiation.cc:299] Negotiation successful
0109 21:03:51.659063 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250109 21:03:51.660064 21355 heartbeater.cc:346] Connected to a master server at 127.12.126.126:35621
I20250109 21:03:51.660189 21355 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:51.660667 21355 heartbeater.cc:510] Master 127.12.126.126:35621 requested a full tablet report, sending...
I20250109 21:03:51.661669 20872 ts_manager.cc:194] Registered new tserver with Master: cb51b5220e604548a6da753363f95292 (127.12.126.67:45411)
I20250109 21:03:51.662427 20872 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:47745
I20250109 21:03:51.665349 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 21224
I20250109 21:03:51.665467 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:03:51.665772 12793 external_mini_cluster.cc:1453] Setting key 4628efed809ddb366b9ac767f3e1ac7f
I20250109 21:03:51.667793 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 21:03:51.668540 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.662181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:56667 (local address 127.12.126.126:35621)
0109 21:03:51.662327 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:51.662332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:51.662490 (+ 158us) server_negotiation.cc:407] Connection header received
0109 21:03:51.662614 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:51.662617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:51.662669 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:51.662745 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:51.663342 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.664008 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.664564 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.664690 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.665520 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:51.665536 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:51.665541 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:51.665568 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:51.667179 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:51.667545 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:51.667549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:51.667551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:51.667596 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:51.667800 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:51.667803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:51.667805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:51.667972 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:51.668059 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:51.668287 (+ 228us) server_negotiation.cc:299] Negotiation successful
0109 21:03:51.668391 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250109 21:03:51.668563 21361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.662142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35621 (local address 127.12.126.67:56667)
0109 21:03:51.662418 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:51.662430 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:51.662524 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:51.662782 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:51.662786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:51.662999 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:51.663221 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.663228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.664123 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.664126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:51.664470 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.664476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.664565 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.665092 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:51.665106 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:51.665381 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:51.667283 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:51.667287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:51.667289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:51.667455 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:51.667676 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:51.667680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:51.667682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:51.667727 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:51.668059 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:51.668062 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:51.668139 (+ 77us) client_negotiation.cc:769] Sending connection context
0109 21:03:51.668229 (+ 90us) client_negotiation.cc:240] Negotiation successful
0109 21:03:51.668376 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":119,"threads_started":1}
Jan 09 21:03:51 dist-test-slave-g0zb krb5kdc[20827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456631, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 09 21:03:51 dist-test-slave-g0zb krb5kdc[20827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:51 dist-test-slave-g0zb krb5kdc[20827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
W20250109 21:03:51.689652 20921 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:51.684319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58760 (local address 127.12.126.126:35621)
0109 21:03:51.684449 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:51.684453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:51.684465 (+ 12us) server_negotiation.cc:407] Connection header received
0109 21:03:51.684513 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:51.684515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:51.684558 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:51.684635 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:51.685378 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.685885 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.686480 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.686605 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.689510 (+ 2905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58760: BlockingRecv error: recv got EOF from 127.0.0.1:58760 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jan 09 21:03:51 dist-test-slave-g0zb krb5kdc[20827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456631, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM
I20250109 21:03:51.698110 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.691045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58770 (local address 127.12.126.126:35621)
0109 21:03:51.691167 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:51.691172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:51.691336 (+ 164us) server_negotiation.cc:407] Connection header received
0109 21:03:51.691472 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:51.691475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:51.691527 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:51.691601 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:51.692294 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.692777 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.693340 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.693568 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.695666 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:51.695682 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:51.695684 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:51.695710 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:51.696903 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:51.697228 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:51.697230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:51.697232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:51.697268 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:51.697421 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:51.697423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:51.697424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:51.697569 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:51.697669 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:51.697816 (+ 147us) server_negotiation.cc:299] Negotiation successful
0109 21:03:51.697926 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250109 21:03:51.701149 20872 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:03:51.703891 20872 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:03:51.716200 21377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.711832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:33389 (local address 127.0.0.1:37198)
0109 21:03:51.712309 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:51.712351 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:51.712486 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:51.713327 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:51.713333 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:51.713355 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:51.713668 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.713679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.714894 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.714898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:51.715720 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.715730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.715835 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.715878 (+ 43us) client_negotiation.cc:769] Sending connection context
0109 21:03:51.715950 (+ 72us) client_negotiation.cc:240] Negotiation successful
0109 21:03:51.716023 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":82,"threads_started":1}
I20250109 21:03:51.716490 21378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.711938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:45411 (local address 127.0.0.1:52182)
0109 21:03:51.712814 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:51.712830 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:51.712914 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:51.713580 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:51.713583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:51.713608 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:51.713845 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.713854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.715215 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.715218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:51.716118 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.716129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.716231 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.716248 (+ 17us) client_negotiation.cc:769] Sending connection context
0109 21:03:51.716309 (+ 61us) client_negotiation.cc:240] Negotiation successful
0109 21:03:51.716365 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"thread_start_us":98,"threads_started":1}
I20250109 21:03:51.717159 21380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.712523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:43869 (local address 127.0.0.1:54426)
0109 21:03:51.713568 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:51.713580 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:51.713654 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:51.714081 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:51.714084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:51.714103 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:51.714308 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.714315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.715658 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.715661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:51.716809 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.716819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.716923 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.716943 (+ 20us) client_negotiation.cc:769] Sending connection context
0109 21:03:51.716992 (+ 49us) client_negotiation.cc:240] Negotiation successful
0109 21:03:51.717046 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"thread_start_us":76,"threads_started":1}
I20250109 21:03:51.717255 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.711990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.12.126.66:33389)
0109 21:03:51.712791 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:51.712797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:51.712820 (+ 23us) server_negotiation.cc:407] Connection header received
0109 21:03:51.712917 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:51.712922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:51.713112 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:51.713315 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:51.713781 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.714724 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.716166 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.716784 (+ 618us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.716962 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:51.717032 (+ 70us) server_negotiation.cc:299] Negotiation successful
0109 21:03:51.717119 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":121,"threads_started":1}
I20250109 21:03:51.717731 21381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.712704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.12.126.67:45411)
0109 21:03:51.713162 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:51.713169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:51.713190 (+ 21us) server_negotiation.cc:407] Connection header received
0109 21:03:51.713269 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:51.713275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:51.713424 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:51.713585 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:51.713969 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.715084 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.716812 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.717324 (+ 512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.717441 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:51.717505 (+ 64us) server_negotiation.cc:299] Negotiation successful
0109 21:03:51.717607 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":93,"threads_started":1}
I20250109 21:03:51.718108 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.713007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.12.126.65:43869)
0109 21:03:51.713311 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:51.713317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:51.713730 (+ 413us) server_negotiation.cc:407] Connection header received
0109 21:03:51.713818 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:51.713821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:51.713944 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:51.714147 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:51.714414 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.715532 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.716945 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.717729 (+ 784us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.717827 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:51.717902 (+ 75us) server_negotiation.cc:299] Negotiation successful
0109 21:03:51.717978 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":94,"threads_started":1}
I20250109 21:03:51.720026 21009 tablet_service.cc:1467] Processing CreateTablet for tablet e2dbfcb25ab548a285de3b9f991ad6fd (DEFAULT_TABLE table=test-table [id=95736421b81347dba5df6330174fd2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:51.720041 21149 tablet_service.cc:1467] Processing CreateTablet for tablet e2dbfcb25ab548a285de3b9f991ad6fd (DEFAULT_TABLE table=test-table [id=95736421b81347dba5df6330174fd2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:51.720041 21289 tablet_service.cc:1467] Processing CreateTablet for tablet e2dbfcb25ab548a285de3b9f991ad6fd (DEFAULT_TABLE table=test-table [id=95736421b81347dba5df6330174fd2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:51.720960 21289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2dbfcb25ab548a285de3b9f991ad6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:51.720960 21149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2dbfcb25ab548a285de3b9f991ad6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:51.720961 21009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2dbfcb25ab548a285de3b9f991ad6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:51.726712 21383 tablet_bootstrap.cc:492] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92: Bootstrap starting.
I20250109 21:03:51.728168 21385 tablet_bootstrap.cc:492] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292: Bootstrap starting.
I20250109 21:03:51.730273 21383 tablet_bootstrap.cc:654] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:51.730686 21385 tablet_bootstrap.cc:654] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:51.731117 21384 tablet_bootstrap.cc:492] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff: Bootstrap starting.
I20250109 21:03:51.731170 21383 log.cc:826] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:51.731477 21385 log.cc:826] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:51.733124 21383 tablet_bootstrap.cc:492] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92: No bootstrap required, opened a new log
I20250109 21:03:51.733229 21385 tablet_bootstrap.cc:492] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292: No bootstrap required, opened a new log
I20250109 21:03:51.733319 21383 ts_tablet_manager.cc:1397] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250109 21:03:51.733361 21385 ts_tablet_manager.cc:1397] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:03:51.733393 21384 tablet_bootstrap.cc:654] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:51.733996 21384 log.cc:826] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:51.735916 21384 tablet_bootstrap.cc:492] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff: No bootstrap required, opened a new log
I20250109 21:03:51.736080 21384 ts_tablet_manager.cc:1397] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:03:51.736485 21383 raft_consensus.cc:357] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } }
I20250109 21:03:51.736485 21385 raft_consensus.cc:357] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } }
I20250109 21:03:51.736691 21383 raft_consensus.cc:383] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:51.736691 21385 raft_consensus.cc:383] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:51.736745 21385 raft_consensus.cc:738] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb51b5220e604548a6da753363f95292, State: Initialized, Role: FOLLOWER
I20250109 21:03:51.736753 21383 raft_consensus.cc:738] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4976a64c48044865ab6c8917922e8f92, State: Initialized, Role: FOLLOWER
I20250109 21:03:51.737170 21385 consensus_queue.cc:260] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } }
I20250109 21:03:51.737170 21383 consensus_queue.cc:260] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } }
I20250109 21:03:51.738037 21383 ts_tablet_manager.cc:1428] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:03:51.738081 21355 heartbeater.cc:502] Master 127.12.126.126:35621 was elected leader, sending a full tablet report...
I20250109 21:03:51.738121 21075 heartbeater.cc:502] Master 127.12.126.126:35621 was elected leader, sending a full tablet report...
W20250109 21:03:51.738956 21076 tablet.cc:2367] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:03:51.738971 21384 raft_consensus.cc:357] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } }
I20250109 21:03:51.739116 21384 raft_consensus.cc:383] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:51.739151 21384 raft_consensus.cc:738] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eec2002b5d9418c84bc049d10f686ff, State: Initialized, Role: FOLLOWER
I20250109 21:03:51.739301 21385 ts_tablet_manager.cc:1428] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250109 21:03:51.739565 21384 consensus_queue.cc:260] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } }
I20250109 21:03:51.740278 21215 heartbeater.cc:502] Master 127.12.126.126:35621 was elected leader, sending a full tablet report...
I20250109 21:03:51.740389 21384 ts_tablet_manager.cc:1428] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250109 21:03:51.818481 21389 raft_consensus.cc:491] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:03:51.818738 21389 raft_consensus.cc:513] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } }
I20250109 21:03:51.819904 21389 leader_election.cc:290] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eec2002b5d9418c84bc049d10f686ff (127.12.126.66:33389), cb51b5220e604548a6da753363f95292 (127.12.126.67:45411)
I20250109 21:03:51.822860 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.820132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:33389 (local address 127.12.126.65:59047)
0109 21:03:51.820465 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:51.820477 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:51.820606 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:51.820848 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:51.820851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:51.820866 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:51.821067 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.821073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.821784 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.821787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:51.822386 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.822394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.822504 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.822537 (+ 33us) client_negotiation.cc:769] Sending connection context
0109 21:03:51.822595 (+ 58us) client_negotiation.cc:240] Negotiation successful
0109 21:03:51.822658 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":87,"threads_started":1}
I20250109 21:03:51.822858 21392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.820160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:45411 (local address 127.12.126.65:32851)
0109 21:03:51.820571 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:51.820582 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:51.820685 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:51.820919 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:51.820922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:51.820936 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:51.821147 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.821153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.821829 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.821832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:51.822412 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.822420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.822533 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.822553 (+ 20us) client_negotiation.cc:769] Sending connection context
0109 21:03:51.822606 (+ 53us) client_negotiation.cc:240] Negotiation successful
0109 21:03:51.822665 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":76,"threads_started":1}
I20250109 21:03:51.823266 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.820219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:59047 (local address 127.12.126.66:33389)
0109 21:03:51.820384 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:51.820388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:51.820551 (+ 163us) server_negotiation.cc:407] Connection header received
0109 21:03:51.820712 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:51.820714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:51.820759 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:51.820828 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:51.821170 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.821670 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.822497 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.822988 (+ 491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.823038 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:51.823102 (+ 64us) server_negotiation.cc:299] Negotiation successful
0109 21:03:51.823161 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20250109 21:03:51.823333 21381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.820311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:32851 (local address 127.12.126.67:45411)
0109 21:03:51.820433 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:51.820437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:51.820658 (+ 221us) server_negotiation.cc:407] Connection header received
0109 21:03:51.820792 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:51.820795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:51.820838 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:51.820904 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:51.821256 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.821730 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.822535 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.823058 (+ 523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.823104 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:51.823158 (+ 54us) server_negotiation.cc:299] Negotiation successful
0109 21:03:51.823241 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20250109 21:03:51.823884 21169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2dbfcb25ab548a285de3b9f991ad6fd" candidate_uuid: "4976a64c48044865ab6c8917922e8f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eec2002b5d9418c84bc049d10f686ff" is_pre_election: true
I20250109 21:03:51.823899 21309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2dbfcb25ab548a285de3b9f991ad6fd" candidate_uuid: "4976a64c48044865ab6c8917922e8f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb51b5220e604548a6da753363f95292" is_pre_election: true
I20250109 21:03:51.824156 21169 raft_consensus.cc:2463] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4976a64c48044865ab6c8917922e8f92 in term 0.
I20250109 21:03:51.824158 21309 raft_consensus.cc:2463] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4976a64c48044865ab6c8917922e8f92 in term 0.
I20250109 21:03:51.824599 20960 leader_election.cc:304] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4976a64c48044865ab6c8917922e8f92, 4eec2002b5d9418c84bc049d10f686ff; no voters:
I20250109 21:03:51.824831 21389 raft_consensus.cc:2798] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:03:51.824901 21389 raft_consensus.cc:491] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:03:51.824946 21389 raft_consensus.cc:3054] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:51.825889 21389 raft_consensus.cc:513] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } }
I20250109 21:03:51.826220 21389 leader_election.cc:290] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [CANDIDATE]: Term 1 election: Requested vote from peers 4eec2002b5d9418c84bc049d10f686ff (127.12.126.66:33389), cb51b5220e604548a6da753363f95292 (127.12.126.67:45411)
I20250109 21:03:51.826563 21169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2dbfcb25ab548a285de3b9f991ad6fd" candidate_uuid: "4976a64c48044865ab6c8917922e8f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eec2002b5d9418c84bc049d10f686ff"
I20250109 21:03:51.826571 21309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2dbfcb25ab548a285de3b9f991ad6fd" candidate_uuid: "4976a64c48044865ab6c8917922e8f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb51b5220e604548a6da753363f95292"
I20250109 21:03:51.826678 21169 raft_consensus.cc:3054] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:51.826690 21309 raft_consensus.cc:3054] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:51.827665 21309 raft_consensus.cc:2463] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4976a64c48044865ab6c8917922e8f92 in term 1.
I20250109 21:03:51.827694 21169 raft_consensus.cc:2463] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4976a64c48044865ab6c8917922e8f92 in term 1.
I20250109 21:03:51.827986 20961 leader_election.cc:304] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4976a64c48044865ab6c8917922e8f92, cb51b5220e604548a6da753363f95292; no voters:
I20250109 21:03:51.828143 21389 raft_consensus.cc:2798] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:51.828337 21389 raft_consensus.cc:695] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [term 1 LEADER]: Becoming Leader. State: Replica: 4976a64c48044865ab6c8917922e8f92, State: Running, Role: LEADER
I20250109 21:03:51.828665 21389 consensus_queue.cc:237] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } }
I20250109 21:03:51.831775 20871 catalog_manager.cc:5526] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4976a64c48044865ab6c8917922e8f92 (127.12.126.65). New cstate: current_term: 1 leader_uuid: "4976a64c48044865ab6c8917922e8f92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } health_report { overall_health: HEALTHY } } }
I20250109 21:03:51.841157 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.838251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.12.126.65:43869)
0109 21:03:51.838383 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:51.838387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:51.838400 (+ 13us) server_negotiation.cc:407] Connection header received
0109 21:03:51.838528 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:51.838530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:51.838585 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:51.838680 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:51.839017 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.839533 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.840129 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.840268 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.840323 (+ 55us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:51.840770 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:51.840883 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:51.840962 (+ 79us) server_negotiation.cc:299] Negotiation successful
0109 21:03:51.841009 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250109 21:03:51.846487 21169 raft_consensus.cc:1270] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff [term 1 FOLLOWER]: Refusing update from remote peer 4976a64c48044865ab6c8917922e8f92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:51.846496 21309 raft_consensus.cc:1270] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292 [term 1 FOLLOWER]: Refusing update from remote peer 4976a64c48044865ab6c8917922e8f92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:51.847301 21394 consensus_queue.cc:1035] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:51.847476 21389 consensus_queue.cc:1035] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:51.854704 21397 mvcc.cc:204] Tried to move back new op lower bound from 7112526364037042176 to 7112526363972874240. Current Snapshot: MvccSnapshot[applied={T|T < 7112526364037042176}]
I20250109 21:03:51.855242 21399 mvcc.cc:204] Tried to move back new op lower bound from 7112526364037042176 to 7112526363972874240. Current Snapshot: MvccSnapshot[applied={T|T < 7112526364037042176}]
I20250109 21:03:51.862434 20871 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58770:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:51.862605 20871 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58770:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:51.864872 20871 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967: Sending DeleteTablet for 3 replicas of tablet e2dbfcb25ab548a285de3b9f991ad6fd
I20250109 21:03:51.865682 21149 tablet_service.cc:1514] Processing DeleteTablet for tablet e2dbfcb25ab548a285de3b9f991ad6fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:51 UTC) from {username='oryx', principal='oryx/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:37198
I20250109 21:03:51.865837 21009 tablet_service.cc:1514] Processing DeleteTablet for tablet e2dbfcb25ab548a285de3b9f991ad6fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:51 UTC) from {username='oryx', principal='oryx/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:54426
I20250109 21:03:51.866075 21289 tablet_service.cc:1514] Processing DeleteTablet for tablet e2dbfcb25ab548a285de3b9f991ad6fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:51 UTC) from {username='oryx', principal='oryx/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:52182
I20250109 21:03:51.866322 21408 tablet_replica.cc:331] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92: stopping tablet replica
I20250109 21:03:51.866520 21409 tablet_replica.cc:331] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292: stopping tablet replica
I20250109 21:03:51.866583 21408 raft_consensus.cc:2238] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:03:51.866756 21409 raft_consensus.cc:2238] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:51.866945 21408 raft_consensus.cc:2267] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:51.867081 21409 raft_consensus.cc:2267] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:51.867287 20871 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:03:51.867745 20871 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:03:51.868314 21408 ts_tablet_manager.cc:1905] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:51.870311 21409 ts_tablet_manager.cc:1905] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:51.871021 21408 ts_tablet_manager.cc:1918] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:03:51.871104 21408 log.cc:1198] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/wal/wals/e2dbfcb25ab548a285de3b9f991ad6fd
I20250109 21:03:51.871454 21408 ts_tablet_manager.cc:1939] T e2dbfcb25ab548a285de3b9f991ad6fd P 4976a64c48044865ab6c8917922e8f92: Deleting consensus metadata
I20250109 21:03:51.872685 20856 catalog_manager.cc:4872] TS 4976a64c48044865ab6c8917922e8f92 (127.12.126.65:43869): tablet e2dbfcb25ab548a285de3b9f991ad6fd (table test-table [id=95736421b81347dba5df6330174fd2c2]) successfully deleted
I20250109 21:03:51.873549 21149 tablet_service.cc:1467] Processing CreateTablet for tablet 8b46f269f46843518f607ccc04827214 (DEFAULT_TABLE table=test-table [id=f348308a340a436f818909305ce4991c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:51.873579 21009 tablet_service.cc:1467] Processing CreateTablet for tablet 8b46f269f46843518f607ccc04827214 (DEFAULT_TABLE table=test-table [id=f348308a340a436f818909305ce4991c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:51.873827 21009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b46f269f46843518f607ccc04827214. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:51.873831 21149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b46f269f46843518f607ccc04827214. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:51.875247 21289 tablet_service.cc:1467] Processing CreateTablet for tablet 8b46f269f46843518f607ccc04827214 (DEFAULT_TABLE table=test-table [id=f348308a340a436f818909305ce4991c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:51.875520 21289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b46f269f46843518f607ccc04827214. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:51.876132 21383 tablet_bootstrap.cc:492] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92: Bootstrap starting.
I20250109 21:03:51.876134 21384 tablet_bootstrap.cc:492] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff: Bootstrap starting.
I20250109 21:03:51.877173 21383 tablet_bootstrap.cc:654] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:51.877183 21384 tablet_bootstrap.cc:654] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:51.877786 21385 tablet_bootstrap.cc:492] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292: Bootstrap starting.
I20250109 21:03:51.878770 21383 tablet_bootstrap.cc:492] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92: No bootstrap required, opened a new log
I20250109 21:03:51.878904 21385 tablet_bootstrap.cc:654] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:51.878903 21383 ts_tablet_manager.cc:1397] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20250109 21:03:51.879410 21383 raft_consensus.cc:357] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:51.879510 21384 tablet_bootstrap.cc:492] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff: No bootstrap required, opened a new log
I20250109 21:03:51.879536 21383 raft_consensus.cc:383] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:51.879572 21383 raft_consensus.cc:738] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4976a64c48044865ab6c8917922e8f92, State: Initialized, Role: FOLLOWER
I20250109 21:03:51.879586 21407 tablet_replica.cc:331] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff: stopping tablet replica
I20250109 21:03:51.879609 21384 ts_tablet_manager.cc:1397] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250109 21:03:51.879699 21383 consensus_queue.cc:260] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:51.879880 21407 raft_consensus.cc:2238] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:51.879979 21383 ts_tablet_manager.cc:1428] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250109 21:03:51.880066 21384 raft_consensus.cc:357] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:51.880193 21384 raft_consensus.cc:383] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:51.880234 21384 raft_consensus.cc:738] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eec2002b5d9418c84bc049d10f686ff, State: Initialized, Role: FOLLOWER
I20250109 21:03:51.880383 21384 consensus_queue.cc:260] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:51.880681 21409 ts_tablet_manager.cc:1918] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:03:51.880703 21384 ts_tablet_manager.cc:1428] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 21:03:51.880239 21407 raft_consensus.cc:2267] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:51.881572 21409 log.cc:1198] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/e2dbfcb25ab548a285de3b9f991ad6fd
I20250109 21:03:51.881939 21409 ts_tablet_manager.cc:1939] T e2dbfcb25ab548a285de3b9f991ad6fd P cb51b5220e604548a6da753363f95292: Deleting consensus metadata
I20250109 21:03:51.883237 20857 catalog_manager.cc:4872] TS cb51b5220e604548a6da753363f95292 (127.12.126.67:45411): tablet e2dbfcb25ab548a285de3b9f991ad6fd (table test-table [id=95736421b81347dba5df6330174fd2c2]) successfully deleted
I20250109 21:03:51.883795 21407 ts_tablet_manager.cc:1905] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:51.886270 21407 ts_tablet_manager.cc:1918] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:03:51.886379 21407 log.cc:1198] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/e2dbfcb25ab548a285de3b9f991ad6fd
I20250109 21:03:51.886684 21407 ts_tablet_manager.cc:1939] T e2dbfcb25ab548a285de3b9f991ad6fd P 4eec2002b5d9418c84bc049d10f686ff: Deleting consensus metadata
I20250109 21:03:51.887523 21385 tablet_bootstrap.cc:492] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292: No bootstrap required, opened a new log
I20250109 21:03:51.887638 21385 ts_tablet_manager.cc:1397] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20250109 21:03:51.887938 20856 catalog_manager.cc:4872] TS 4eec2002b5d9418c84bc049d10f686ff (127.12.126.66:33389): tablet e2dbfcb25ab548a285de3b9f991ad6fd (table test-table [id=95736421b81347dba5df6330174fd2c2]) successfully deleted
I20250109 21:03:51.888157 21385 raft_consensus.cc:357] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:51.888263 21385 raft_consensus.cc:383] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:51.888311 21385 raft_consensus.cc:738] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb51b5220e604548a6da753363f95292, State: Initialized, Role: FOLLOWER
I20250109 21:03:51.888468 21385 consensus_queue.cc:260] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:51.888743 21385 ts_tablet_manager.cc:1428] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20250109 21:03:51.908253 21356 tablet.cc:2367] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:03:51.940778 21391 raft_consensus.cc:491] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:03:51.940941 21391 raft_consensus.cc:513] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:51.942320 21391 leader_election.cc:290] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4976a64c48044865ab6c8917922e8f92 (127.12.126.65:43869), cb51b5220e604548a6da753363f95292 (127.12.126.67:45411)
I20250109 21:03:51.945785 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.942516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:43869 (local address 127.12.126.66:58783)
0109 21:03:51.943053 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:51.943067 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:51.943171 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:51.943562 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:51.943565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:51.943585 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:51.943865 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.943875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.944797 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.944800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:51.945344 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.945352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.945467 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.945490 (+ 23us) client_negotiation.cc:769] Sending connection context
0109 21:03:51.945557 (+ 67us) client_negotiation.cc:240] Negotiation successful
0109 21:03:51.945618 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":13,"thread_start_us":86,"threads_started":1}
I20250109 21:03:51.945827 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.942516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:45411 (local address 127.12.126.66:57065)
0109 21:03:51.942712 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:51.942727 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:51.942852 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:51.943396 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:51.943398 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:51.943416 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:51.943635 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.943642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.944521 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.944524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:51.945461 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:51.945471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.945579 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.945598 (+ 19us) client_negotiation.cc:769] Sending connection context
0109 21:03:51.945647 (+ 49us) client_negotiation.cc:240] Negotiation successful
0109 21:03:51.945707 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20250109 21:03:51.946153 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.942846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:58783 (local address 127.12.126.65:43869)
0109 21:03:51.942977 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:51.942982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:51.943154 (+ 172us) server_negotiation.cc:407] Connection header received
0109 21:03:51.943291 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:51.943294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:51.943345 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:51.943421 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:51.943989 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.944659 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.945457 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.945900 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.945947 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:51.946004 (+ 57us) server_negotiation.cc:299] Negotiation successful
0109 21:03:51.946065 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250109 21:03:51.946789 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b46f269f46843518f607ccc04827214" candidate_uuid: "4eec2002b5d9418c84bc049d10f686ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4976a64c48044865ab6c8917922e8f92" is_pre_election: true
I20250109 21:03:51.947050 21381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:51.943027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:57065 (local address 127.12.126.67:45411)
0109 21:03:51.943175 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:51.943180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:51.943196 (+ 16us) server_negotiation.cc:407] Connection header received
0109 21:03:51.943254 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:51.943258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:51.943308 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:51.943378 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:51.943751 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.944408 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:51.946248 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:51.946758 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:51.946795 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:51.946874 (+ 79us) server_negotiation.cc:299] Negotiation successful
0109 21:03:51.946934 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20250109 21:03:51.947052 21029 raft_consensus.cc:2463] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eec2002b5d9418c84bc049d10f686ff in term 0.
I20250109 21:03:51.947423 21309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b46f269f46843518f607ccc04827214" candidate_uuid: "4eec2002b5d9418c84bc049d10f686ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb51b5220e604548a6da753363f95292" is_pre_election: true
I20250109 21:03:51.947460 21100 leader_election.cc:304] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4976a64c48044865ab6c8917922e8f92, 4eec2002b5d9418c84bc049d10f686ff; no voters:
I20250109 21:03:51.947569 21309 raft_consensus.cc:2463] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eec2002b5d9418c84bc049d10f686ff in term 0.
I20250109 21:03:51.947736 21391 raft_consensus.cc:2798] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:03:51.947795 21391 raft_consensus.cc:491] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:03:51.947844 21391 raft_consensus.cc:3054] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Advancing to term 1
W20250109 21:03:51.948148 21216 tablet.cc:2367] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:03:51.948921 21391 raft_consensus.cc:513] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:51.949265 21391 leader_election.cc:290] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [CANDIDATE]: Term 1 election: Requested vote from peers 4976a64c48044865ab6c8917922e8f92 (127.12.126.65:43869), cb51b5220e604548a6da753363f95292 (127.12.126.67:45411)
I20250109 21:03:51.949644 21309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b46f269f46843518f607ccc04827214" candidate_uuid: "4eec2002b5d9418c84bc049d10f686ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb51b5220e604548a6da753363f95292"
I20250109 21:03:51.949664 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b46f269f46843518f607ccc04827214" candidate_uuid: "4eec2002b5d9418c84bc049d10f686ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4976a64c48044865ab6c8917922e8f92"
I20250109 21:03:51.949762 21309 raft_consensus.cc:3054] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:51.949775 21029 raft_consensus.cc:3054] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:51.950605 21309 raft_consensus.cc:2463] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eec2002b5d9418c84bc049d10f686ff in term 1.
I20250109 21:03:51.950922 21101 leader_election.cc:304] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eec2002b5d9418c84bc049d10f686ff, cb51b5220e604548a6da753363f95292; no voters:
I20250109 21:03:51.951092 21391 raft_consensus.cc:2798] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:51.951268 21391 raft_consensus.cc:695] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [term 1 LEADER]: Becoming Leader. State: Replica: 4eec2002b5d9418c84bc049d10f686ff, State: Running, Role: LEADER
I20250109 21:03:51.951375 21029 raft_consensus.cc:2463] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eec2002b5d9418c84bc049d10f686ff in term 1.
I20250109 21:03:51.951545 21391 consensus_queue.cc:237] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:51.953662 20870 catalog_manager.cc:5526] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eec2002b5d9418c84bc049d10f686ff (127.12.126.66). New cstate: current_term: 1 leader_uuid: "4eec2002b5d9418c84bc049d10f686ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } health_report { overall_health: UNKNOWN } } }
Jan 09 21:03:51 dist-test-slave-g0zb krb5kdc[20827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456630, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.126@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM
I20250109 21:03:52.009495 21421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.000978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35621 (local address 127.0.0.1:58778)
0109 21:03:52.001374 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:52.001387 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:52.001490 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:52.001763 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:52.001765 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:52.002151 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:52.002363 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:52.002370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.003391 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.003393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:52.003833 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:52.003842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.003937 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.004508 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:52.004522 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:52.006071 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:52.008134 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:52.008141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:52.008147 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:52.008375 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:52.008630 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:52.008632 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:52.008634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:52.008673 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:52.009051 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:52.009056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:52.009148 (+ 92us) client_negotiation.cc:769] Sending connection context
0109 21:03:52.009231 (+ 83us) client_negotiation.cc:240] Negotiation successful
0109 21:03:52.009346 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":120,"threads_started":1}
I20250109 21:03:52.009627 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.001068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58778 (local address 127.12.126.126:35621)
0109 21:03:52.001221 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:52.001226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:52.001472 (+ 246us) server_negotiation.cc:407] Connection header received
0109 21:03:52.001599 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:52.001603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:52.001659 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:52.001749 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:52.002499 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.003289 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.003952 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.004137 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.006207 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:52.006227 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:52.006230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:52.006256 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:52.008026 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:52.008484 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:52.008488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:52.008491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:52.008543 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:52.008760 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:52.008764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:52.008766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:52.008958 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:52.009064 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:52.009315 (+ 251us) server_negotiation.cc:299] Negotiation successful
0109 21:03:52.009434 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250109 21:03:52.013778 20870 catalog_manager.cc:1909] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:58778:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250109 21:03:52.014537 20870 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:03:52.019780 21009 tablet_service.cc:1467] Processing CreateTablet for tablet 47b41d1f8d25417bbdee5dbaa1776e70 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7a4e8e33c5a0482395943d3f7d605252]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250109 21:03:52.019871 21149 tablet_service.cc:1467] Processing CreateTablet for tablet 47b41d1f8d25417bbdee5dbaa1776e70 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7a4e8e33c5a0482395943d3f7d605252]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250109 21:03:52.019994 21289 tablet_service.cc:1467] Processing CreateTablet for tablet 47b41d1f8d25417bbdee5dbaa1776e70 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7a4e8e33c5a0482395943d3f7d605252]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250109 21:03:52.020205 21009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47b41d1f8d25417bbdee5dbaa1776e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:52.020239 21149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47b41d1f8d25417bbdee5dbaa1776e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:52.020292 21289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47b41d1f8d25417bbdee5dbaa1776e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:52.022917 21383 tablet_bootstrap.cc:492] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92: Bootstrap starting.
I20250109 21:03:52.022970 21384 tablet_bootstrap.cc:492] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff: Bootstrap starting.
I20250109 21:03:52.023018 21385 tablet_bootstrap.cc:492] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292: Bootstrap starting.
I20250109 21:03:52.024001 21385 tablet_bootstrap.cc:654] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:52.024001 21384 tablet_bootstrap.cc:654] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:52.024199 21383 tablet_bootstrap.cc:654] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:52.025801 21383 tablet_bootstrap.cc:492] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92: No bootstrap required, opened a new log
I20250109 21:03:52.025905 21383 ts_tablet_manager.cc:1397] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250109 21:03:52.026479 21383 raft_consensus.cc:357] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.026597 21383 raft_consensus.cc:383] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:52.026633 21383 raft_consensus.cc:738] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4976a64c48044865ab6c8917922e8f92, State: Initialized, Role: FOLLOWER
I20250109 21:03:52.026707 21385 tablet_bootstrap.cc:492] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292: No bootstrap required, opened a new log
I20250109 21:03:52.026722 21384 tablet_bootstrap.cc:492] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff: No bootstrap required, opened a new log
I20250109 21:03:52.026793 21384 ts_tablet_manager.cc:1397] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250109 21:03:52.026788 21383 consensus_queue.cc:260] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.027082 21404 tablet_replica.cc:440] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } }
I20250109 21:03:52.027221 21404 tablet_replica.cc:443] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:52.027076 21383 ts_tablet_manager.cc:1428] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250109 21:03:52.027295 21384 raft_consensus.cc:357] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.027384 21384 raft_consensus.cc:383] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:52.027415 21384 raft_consensus.cc:738] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eec2002b5d9418c84bc049d10f686ff, State: Initialized, Role: FOLLOWER
I20250109 21:03:52.027541 21384 consensus_queue.cc:260] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.026785 21385 ts_tablet_manager.cc:1397] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250109 21:03:52.027837 21384 ts_tablet_manager.cc:1428] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 21:03:52.027890 21391 tablet_replica.cc:440] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } }
I20250109 21:03:52.027990 21391 tablet_replica.cc:443] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:52.028141 21385 raft_consensus.cc:357] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.028246 21385 raft_consensus.cc:383] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:52.028281 21385 raft_consensus.cc:738] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb51b5220e604548a6da753363f95292, State: Initialized, Role: FOLLOWER
I20250109 21:03:52.028403 21385 consensus_queue.cc:260] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.028662 21385 ts_tablet_manager.cc:1428] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 21:03:52.028904 21390 tablet_replica.cc:440] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } }
I20250109 21:03:52.029011 21390 tablet_replica.cc:443] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:52.152189 21404 raft_consensus.cc:491] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:03:52.152336 21404 raft_consensus.cc:513] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.152792 21404 leader_election.cc:290] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eec2002b5d9418c84bc049d10f686ff (127.12.126.66:33389), cb51b5220e604548a6da753363f95292 (127.12.126.67:45411)
I20250109 21:03:52.153234 21169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b41d1f8d25417bbdee5dbaa1776e70" candidate_uuid: "4976a64c48044865ab6c8917922e8f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eec2002b5d9418c84bc049d10f686ff" is_pre_election: true
I20250109 21:03:52.153266 21309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b41d1f8d25417bbdee5dbaa1776e70" candidate_uuid: "4976a64c48044865ab6c8917922e8f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb51b5220e604548a6da753363f95292" is_pre_election: true
I20250109 21:03:52.153370 21169 raft_consensus.cc:2463] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4976a64c48044865ab6c8917922e8f92 in term 0.
I20250109 21:03:52.153447 21309 raft_consensus.cc:2463] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4976a64c48044865ab6c8917922e8f92 in term 0.
I20250109 21:03:52.153693 20960 leader_election.cc:304] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4976a64c48044865ab6c8917922e8f92, 4eec2002b5d9418c84bc049d10f686ff; no voters:
I20250109 21:03:52.153869 21404 raft_consensus.cc:2798] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:03:52.153939 21404 raft_consensus.cc:491] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:03:52.153993 21404 raft_consensus.cc:3054] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:52.155104 21404 raft_consensus.cc:513] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.155462 21404 leader_election.cc:290] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [CANDIDATE]: Term 1 election: Requested vote from peers 4eec2002b5d9418c84bc049d10f686ff (127.12.126.66:33389), cb51b5220e604548a6da753363f95292 (127.12.126.67:45411)
I20250109 21:03:52.155838 21169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b41d1f8d25417bbdee5dbaa1776e70" candidate_uuid: "4976a64c48044865ab6c8917922e8f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eec2002b5d9418c84bc049d10f686ff"
I20250109 21:03:52.155851 21309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b41d1f8d25417bbdee5dbaa1776e70" candidate_uuid: "4976a64c48044865ab6c8917922e8f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb51b5220e604548a6da753363f95292"
I20250109 21:03:52.155969 21309 raft_consensus.cc:3054] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:52.155969 21169 raft_consensus.cc:3054] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:52.156996 21309 raft_consensus.cc:2463] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4976a64c48044865ab6c8917922e8f92 in term 1.
I20250109 21:03:52.156996 21169 raft_consensus.cc:2463] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4976a64c48044865ab6c8917922e8f92 in term 1.
I20250109 21:03:52.157306 20961 leader_election.cc:304] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4976a64c48044865ab6c8917922e8f92, cb51b5220e604548a6da753363f95292; no voters:
I20250109 21:03:52.157486 21404 raft_consensus.cc:2798] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:52.157570 21404 raft_consensus.cc:695] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [term 1 LEADER]: Becoming Leader. State: Replica: 4976a64c48044865ab6c8917922e8f92, State: Running, Role: LEADER
I20250109 21:03:52.157738 21404 consensus_queue.cc:237] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.158393 21394 tablet_replica.cc:440] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92: TxnStatusTablet state changed. Reason: New leader 4976a64c48044865ab6c8917922e8f92. Latest consensus state: current_term: 1 leader_uuid: "4976a64c48044865ab6c8917922e8f92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } }
I20250109 21:03:52.158499 21394 tablet_replica.cc:443] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92: This TxnStatusTablet replica's current role is: LEADER
I20250109 21:03:52.158764 21426 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250109 21:03:52.159001 21426 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250109 21:03:52.159679 20871 catalog_manager.cc:5526] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4976a64c48044865ab6c8917922e8f92 (127.12.126.65). New cstate: current_term: 1 leader_uuid: "4976a64c48044865ab6c8917922e8f92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } health_report { overall_health: UNKNOWN } } }
I20250109 21:03:52.187098 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.184279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54448 (local address 127.12.126.65:43869)
0109 21:03:52.184401 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:52.184405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:52.184520 (+ 115us) server_negotiation.cc:407] Connection header received
0109 21:03:52.184708 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:52.184711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:52.184758 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:52.184828 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:52.185235 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.185698 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.186538 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.186653 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.186691 (+ 38us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:52.186759 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:52.186848 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:52.186943 (+ 95us) server_negotiation.cc:299] Negotiation successful
0109 21:03:52.186987 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250109 21:03:52.187139 21421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.184175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:43869 (local address 127.0.0.1:54448)
0109 21:03:52.184437 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:52.184453 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:52.184591 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:52.184868 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:52.184871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:52.184879 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0109 21:03:52.185133 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:52.185140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.185793 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.185796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:52.186430 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:52.186439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.186554 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.186604 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0109 21:03:52.186848 (+ 244us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0109 21:03:52.186863 (+ 15us) client_negotiation.cc:769] Sending connection context
0109 21:03:52.186930 (+ 67us) client_negotiation.cc:240] Negotiation successful
0109 21:03:52.187003 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20250109 21:03:52.189640 21169 raft_consensus.cc:1270] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff [term 1 FOLLOWER]: Refusing update from remote peer 4976a64c48044865ab6c8917922e8f92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:52.189640 21309 raft_consensus.cc:1270] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292 [term 1 FOLLOWER]: Refusing update from remote peer 4976a64c48044865ab6c8917922e8f92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:52.190066 21404 consensus_queue.cc:1035] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:52.190202 21394 consensus_queue.cc:1035] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:52.191582 21390 tablet_replica.cc:440] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292: TxnStatusTablet state changed. Reason: New leader 4976a64c48044865ab6c8917922e8f92. Latest consensus state: current_term: 1 leader_uuid: "4976a64c48044865ab6c8917922e8f92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } }
I20250109 21:03:52.191711 21390 tablet_replica.cc:443] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:52.191533 21391 tablet_replica.cc:440] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff: TxnStatusTablet state changed. Reason: New leader 4976a64c48044865ab6c8917922e8f92. Latest consensus state: current_term: 1 leader_uuid: "4976a64c48044865ab6c8917922e8f92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } }
I20250109 21:03:52.192083 21391 tablet_replica.cc:443] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:52.194106 21404 tablet_replica.cc:440] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4976a64c48044865ab6c8917922e8f92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } }
I20250109 21:03:52.194233 21404 tablet_replica.cc:443] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92: This TxnStatusTablet replica's current role is: LEADER
I20250109 21:03:52.196795 21390 tablet_replica.cc:440] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4976a64c48044865ab6c8917922e8f92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } }
I20250109 21:03:52.196915 21390 tablet_replica.cc:443] T 47b41d1f8d25417bbdee5dbaa1776e70 P cb51b5220e604548a6da753363f95292: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:52.197966 21394 tablet_replica.cc:440] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4976a64c48044865ab6c8917922e8f92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } }
I20250109 21:03:52.198071 21394 tablet_replica.cc:443] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4976a64c48044865ab6c8917922e8f92: This TxnStatusTablet replica's current role is: LEADER
I20250109 21:03:52.199011 21391 tablet_replica.cc:440] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4976a64c48044865ab6c8917922e8f92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } } }
I20250109 21:03:52.199119 21391 tablet_replica.cc:443] T 47b41d1f8d25417bbdee5dbaa1776e70 P 4eec2002b5d9418c84bc049d10f686ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20250109 21:03:52.203143 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.199610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37200 (local address 127.12.126.66:33389)
0109 21:03:52.199838 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:52.199843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:52.199855 (+ 12us) server_negotiation.cc:407] Connection header received
0109 21:03:52.199942 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:52.199945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:52.200014 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:52.200127 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:52.200586 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.201307 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.202009 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.202196 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.202289 (+ 93us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:52.202747 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:52.202911 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:52.202956 (+ 45us) server_negotiation.cc:299] Negotiation successful
0109 21:03:52.203007 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20250109 21:03:52.212734 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.209927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:39539 (local address 127.12.126.65:43869)
0109 21:03:52.210153 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:52.210162 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:52.210201 (+ 39us) server_negotiation.cc:407] Connection header received
0109 21:03:52.210370 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:52.210373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:52.210417 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:52.210482 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:52.210912 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.211376 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.212051 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.212244 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.212288 (+ 44us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:52.212355 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:52.212421 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:52.212554 (+ 133us) server_negotiation.cc:299] Negotiation successful
0109 21:03:52.212600 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20250109 21:03:52.212764 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.209830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:43869 (local address 127.12.126.66:39539)
0109 21:03:52.210127 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:52.210142 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:52.210256 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:52.210496 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:52.210500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:52.210507 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0109 21:03:52.210752 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:52.210757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.211507 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.211510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:52.211942 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:52.211950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.212055 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.212101 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0109 21:03:52.212448 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0109 21:03:52.212456 (+ 8us) client_negotiation.cc:769] Sending connection context
0109 21:03:52.212517 (+ 61us) client_negotiation.cc:240] Negotiation successful
0109 21:03:52.212600 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
I20250109 21:03:52.219787 21309 raft_consensus.cc:1270] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292 [term 1 FOLLOWER]: Refusing update from remote peer 4eec2002b5d9418c84bc049d10f686ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:52.219874 21029 raft_consensus.cc:1270] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92 [term 1 FOLLOWER]: Refusing update from remote peer 4eec2002b5d9418c84bc049d10f686ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:52.220315 21391 consensus_queue.cc:1035] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:52.220664 21391 consensus_queue.cc:1035] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:52.230691 21398 mvcc.cc:204] Tried to move back new op lower bound from 7112526365567598592 to 7112526364476067840. Current Snapshot: MvccSnapshot[applied={T|T < 7112526365567598592}]
I20250109 21:03:52.249260 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.242613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:39851 (local address 127.12.126.66:33389)
0109 21:03:52.242734 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:52.242738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:52.245704 (+ 2966us) server_negotiation.cc:407] Connection header received
0109 21:03:52.245908 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:52.245911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:52.245967 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:52.246014 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:52.246558 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.247307 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.248254 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.248451 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.248752 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:52.248871 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:52.248916 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:52.249069 (+ 153us) server_negotiation.cc:299] Negotiation successful
0109 21:03:52.249123 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250109 21:03:52.249681 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.242249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:33389 (local address 127.12.126.65:39851)
0109 21:03:52.245615 (+ 3366us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:52.245633 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:52.245783 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:52.246144 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:52.246147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:52.246154 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0109 21:03:52.246418 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:52.246425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.247433 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.247438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:52.248115 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:52.248124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.248538 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.248585 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0109 21:03:52.248975 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0109 21:03:52.248981 (+ 6us) client_negotiation.cc:769] Sending connection context
0109 21:03:52.249458 (+ 477us) client_negotiation.cc:240] Negotiation successful
0109 21:03:52.249545 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3216,"thread_start_us":113,"threads_started":1}
I20250109 21:03:52.281352 20870 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58770:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:52.281544 20870 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58770:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:52.283991 20870 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967: Sending DeleteTablet for 3 replicas of tablet 8b46f269f46843518f607ccc04827214
I20250109 21:03:52.284914 21149 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b46f269f46843518f607ccc04827214 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:52 UTC) from {username='oryx', principal='oryx/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:37198
I20250109 21:03:52.285106 21407 tablet_replica.cc:331] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff: stopping tablet replica
I20250109 21:03:52.285069 21289 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b46f269f46843518f607ccc04827214 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:52 UTC) from {username='oryx', principal='oryx/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:52182
I20250109 21:03:52.285202 21407 raft_consensus.cc:2238] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:03:52.285292 21409 tablet_replica.cc:331] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292: stopping tablet replica
I20250109 21:03:52.285392 21409 raft_consensus.cc:2238] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:52.285513 21409 raft_consensus.cc:2267] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:52.285511 21407 raft_consensus.cc:2267] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:52.286305 21409 ts_tablet_manager.cc:1905] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:52.286321 21407 ts_tablet_manager.cc:1905] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:52.289084 21409 ts_tablet_manager.cc:1918] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250109 21:03:52.289160 21409 log.cc:1198] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/8b46f269f46843518f607ccc04827214
I20250109 21:03:52.289402 21409 ts_tablet_manager.cc:1939] T 8b46f269f46843518f607ccc04827214 P cb51b5220e604548a6da753363f95292: Deleting consensus metadata
I20250109 21:03:52.289391 21407 ts_tablet_manager.cc:1918] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250109 21:03:52.289469 21407 log.cc:1198] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/8b46f269f46843518f607ccc04827214
I20250109 21:03:52.289633 21009 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b46f269f46843518f607ccc04827214 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:52 UTC) from {username='oryx', principal='oryx/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:54426
I20250109 21:03:52.289732 21407 ts_tablet_manager.cc:1939] T 8b46f269f46843518f607ccc04827214 P 4eec2002b5d9418c84bc049d10f686ff: Deleting consensus metadata
I20250109 21:03:52.289919 21408 tablet_replica.cc:331] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92: stopping tablet replica
I20250109 21:03:52.290071 21408 raft_consensus.cc:2238] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:52.290206 21408 raft_consensus.cc:2267] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:52.290354 20857 catalog_manager.cc:4872] TS cb51b5220e604548a6da753363f95292 (127.12.126.67:45411): tablet 8b46f269f46843518f607ccc04827214 (table test-table [id=f348308a340a436f818909305ce4991c]) successfully deleted
I20250109 21:03:52.291143 21408 ts_tablet_manager.cc:1905] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:52.293043 21408 ts_tablet_manager.cc:1918] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250109 21:03:52.293113 21408 log.cc:1198] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/wal/wals/8b46f269f46843518f607ccc04827214
I20250109 21:03:52.293354 21408 ts_tablet_manager.cc:1939] T 8b46f269f46843518f607ccc04827214 P 4976a64c48044865ab6c8917922e8f92: Deleting consensus metadata
I20250109 21:03:52.293906 20856 catalog_manager.cc:4872] TS 4eec2002b5d9418c84bc049d10f686ff (127.12.126.66:33389): tablet 8b46f269f46843518f607ccc04827214 (table test-table [id=f348308a340a436f818909305ce4991c]) successfully deleted
I20250109 21:03:52.294121 20856 catalog_manager.cc:4872] TS 4976a64c48044865ab6c8917922e8f92 (127.12.126.65:43869): tablet 8b46f269f46843518f607ccc04827214 (table test-table [id=f348308a340a436f818909305ce4991c]) successfully deleted
Jan 09 21:03:52 dist-test-slave-g0zb krb5kdc[20827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 09 21:03:52 dist-test-slave-g0zb krb5kdc[20827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:52 dist-test-slave-g0zb krb5kdc[20827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
W20250109 21:03:52.319845 20921 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:52.309073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.12.126.126:35621)
0109 21:03:52.309223 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:52.309228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:52.309427 (+ 199us) server_negotiation.cc:407] Connection header received
0109 21:03:52.309576 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:52.309579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:52.309625 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:52.309702 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:52.310507 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.311066 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.311701 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.312289 (+ 588us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.319709 (+ 7420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58788: BlockingRecv error: recv got EOF from 127.0.0.1:58788 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jan 09 21:03:52 dist-test-slave-g0zb krb5kdc[20827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM
I20250109 21:03:52.329283 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.321439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.12.126.126:35621)
0109 21:03:52.321575 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:52.321582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:52.321971 (+ 389us) server_negotiation.cc:407] Connection header received
0109 21:03:52.322093 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:52.322097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:52.322147 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:52.322228 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:52.322992 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.323554 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.324125 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.324653 (+ 528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.326279 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:52.326295 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:52.326298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:52.326324 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:52.327740 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:52.328228 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:52.328231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:52.328232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:52.328278 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:52.328490 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:52.328493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:52.328495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:52.328654 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:52.328785 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:52.328997 (+ 212us) server_negotiation.cc:299] Negotiation successful
0109 21:03:52.329106 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250109 21:03:52.332298 20870 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:03:52.332772 20870 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:03:52.337577 21009 tablet_service.cc:1467] Processing CreateTablet for tablet cf536da0ea4d4899966ca91671dfdae2 (DEFAULT_TABLE table=test-table [id=5838fdd8b5ad4fada785722c5d8ab060]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:52.337898 21009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf536da0ea4d4899966ca91671dfdae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:52.339816 21383 tablet_bootstrap.cc:492] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92: Bootstrap starting.
I20250109 21:03:52.340224 21289 tablet_service.cc:1467] Processing CreateTablet for tablet cf536da0ea4d4899966ca91671dfdae2 (DEFAULT_TABLE table=test-table [id=5838fdd8b5ad4fada785722c5d8ab060]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:52.340498 21289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf536da0ea4d4899966ca91671dfdae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:52.340796 21383 tablet_bootstrap.cc:654] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:52.342448 21385 tablet_bootstrap.cc:492] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292: Bootstrap starting.
I20250109 21:03:52.342929 21149 tablet_service.cc:1467] Processing CreateTablet for tablet cf536da0ea4d4899966ca91671dfdae2 (DEFAULT_TABLE table=test-table [id=5838fdd8b5ad4fada785722c5d8ab060]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:52.343210 21149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf536da0ea4d4899966ca91671dfdae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:52.343425 21385 tablet_bootstrap.cc:654] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:52.344931 21385 tablet_bootstrap.cc:492] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292: No bootstrap required, opened a new log
I20250109 21:03:52.345012 21385 ts_tablet_manager.cc:1397] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250109 21:03:52.345500 21385 raft_consensus.cc:357] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.345603 21385 raft_consensus.cc:383] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:52.345636 21385 raft_consensus.cc:738] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb51b5220e604548a6da753363f95292, State: Initialized, Role: FOLLOWER
I20250109 21:03:52.345768 21385 consensus_queue.cc:260] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.346029 21385 ts_tablet_manager.cc:1428] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250109 21:03:52.346300 21384 tablet_bootstrap.cc:492] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff: Bootstrap starting.
I20250109 21:03:52.347173 21383 tablet_bootstrap.cc:492] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92: No bootstrap required, opened a new log
I20250109 21:03:52.347285 21383 ts_tablet_manager.cc:1397] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20250109 21:03:52.347322 21384 tablet_bootstrap.cc:654] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:52.347810 21383 raft_consensus.cc:357] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.347918 21383 raft_consensus.cc:383] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:52.347952 21383 raft_consensus.cc:738] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4976a64c48044865ab6c8917922e8f92, State: Initialized, Role: FOLLOWER
I20250109 21:03:52.348073 21383 consensus_queue.cc:260] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.348325 21383 ts_tablet_manager.cc:1428] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 21:03:52.351044 21384 tablet_bootstrap.cc:492] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff: No bootstrap required, opened a new log
I20250109 21:03:52.351153 21384 ts_tablet_manager.cc:1397] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250109 21:03:52.351559 21384 raft_consensus.cc:357] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.351678 21384 raft_consensus.cc:383] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:52.351711 21384 raft_consensus.cc:738] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eec2002b5d9418c84bc049d10f686ff, State: Initialized, Role: FOLLOWER
I20250109 21:03:52.351819 21384 consensus_queue.cc:260] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.352051 21384 ts_tablet_manager.cc:1428] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 21:03:52.726722 21390 raft_consensus.cc:491] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:03:52.726912 21390 raft_consensus.cc:513] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.728020 21390 leader_election.cc:290] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4976a64c48044865ab6c8917922e8f92 (127.12.126.65:43869), 4eec2002b5d9418c84bc049d10f686ff (127.12.126.66:33389)
I20250109 21:03:52.731209 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.728623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:33389 (local address 127.12.126.67:36931)
0109 21:03:52.728828 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:52.728841 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:52.728923 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:52.729149 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:52.729151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:52.729166 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:52.729370 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:52.729376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.730111 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.730115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:52.730759 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:52.730768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.730901 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.730927 (+ 26us) client_negotiation.cc:769] Sending connection context
0109 21:03:52.730982 (+ 55us) client_negotiation.cc:240] Negotiation successful
0109 21:03:52.731048 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":50,"threads_started":1}
I20250109 21:03:52.731273 21456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.728269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:43869 (local address 127.12.126.67:54531)
0109 21:03:52.728537 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:52.728556 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:52.728682 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:52.729217 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:52.729220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:52.729253 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:03:52.729428 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:52.729437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.730268 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.730271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:52.730842 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:52.730852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.730989 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.731009 (+ 20us) client_negotiation.cc:769] Sending connection context
0109 21:03:52.731062 (+ 53us) client_negotiation.cc:240] Negotiation successful
0109 21:03:52.731118 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
I20250109 21:03:52.731642 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.728764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:36931 (local address 127.12.126.66:33389)
0109 21:03:52.728890 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:52.728893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:52.728905 (+ 12us) server_negotiation.cc:407] Connection header received
0109 21:03:52.729019 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:52.729021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:52.729063 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:52.729129 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:52.729509 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.730005 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.730922 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.731390 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.731433 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:52.731493 (+ 60us) server_negotiation.cc:299] Negotiation successful
0109 21:03:52.731542 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250109 21:03:52.731869 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.728372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:54531 (local address 127.12.126.65:43869)
0109 21:03:52.728662 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:52.728670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:52.728691 (+ 21us) server_negotiation.cc:407] Connection header received
0109 21:03:52.728842 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:52.728847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:52.728912 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:52.729002 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:03:52.729583 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.730155 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.730969 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.731606 (+ 637us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.731638 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:52.731685 (+ 47us) server_negotiation.cc:299] Negotiation successful
0109 21:03:52.731754 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":89,"threads_started":1}
I20250109 21:03:52.732038 21169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf536da0ea4d4899966ca91671dfdae2" candidate_uuid: "cb51b5220e604548a6da753363f95292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eec2002b5d9418c84bc049d10f686ff" is_pre_election: true
I20250109 21:03:52.732175 21169 raft_consensus.cc:2463] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb51b5220e604548a6da753363f95292 in term 0.
I20250109 21:03:52.732260 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf536da0ea4d4899966ca91671dfdae2" candidate_uuid: "cb51b5220e604548a6da753363f95292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4976a64c48044865ab6c8917922e8f92" is_pre_election: true
I20250109 21:03:52.732406 21029 raft_consensus.cc:2463] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb51b5220e604548a6da753363f95292 in term 0.
I20250109 21:03:52.732606 21240 leader_election.cc:304] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eec2002b5d9418c84bc049d10f686ff, cb51b5220e604548a6da753363f95292; no voters:
I20250109 21:03:52.732816 21390 raft_consensus.cc:2798] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:03:52.732872 21390 raft_consensus.cc:491] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:03:52.732899 21390 raft_consensus.cc:3054] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:52.733800 21390 raft_consensus.cc:513] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.734127 21390 leader_election.cc:290] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [CANDIDATE]: Term 1 election: Requested vote from peers 4976a64c48044865ab6c8917922e8f92 (127.12.126.65:43869), 4eec2002b5d9418c84bc049d10f686ff (127.12.126.66:33389)
I20250109 21:03:52.734452 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf536da0ea4d4899966ca91671dfdae2" candidate_uuid: "cb51b5220e604548a6da753363f95292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4976a64c48044865ab6c8917922e8f92"
I20250109 21:03:52.734575 21029 raft_consensus.cc:3054] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:52.734545 21169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf536da0ea4d4899966ca91671dfdae2" candidate_uuid: "cb51b5220e604548a6da753363f95292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eec2002b5d9418c84bc049d10f686ff"
I20250109 21:03:52.734663 21169 raft_consensus.cc:3054] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:52.735559 21029 raft_consensus.cc:2463] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb51b5220e604548a6da753363f95292 in term 1.
I20250109 21:03:52.735612 21169 raft_consensus.cc:2463] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb51b5220e604548a6da753363f95292 in term 1.
I20250109 21:03:52.735834 21240 leader_election.cc:304] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4976a64c48044865ab6c8917922e8f92, cb51b5220e604548a6da753363f95292; no voters:
I20250109 21:03:52.735983 21390 raft_consensus.cc:2798] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:52.736152 21390 raft_consensus.cc:695] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [term 1 LEADER]: Becoming Leader. State: Replica: cb51b5220e604548a6da753363f95292, State: Running, Role: LEADER
I20250109 21:03:52.736404 21390 consensus_queue.cc:237] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.738596 20872 catalog_manager.cc:5526] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb51b5220e604548a6da753363f95292 (127.12.126.67). New cstate: current_term: 1 leader_uuid: "cb51b5220e604548a6da753363f95292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } health_report { overall_health: HEALTHY } } }
I20250109 21:03:52.806336 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.803150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52184 (local address 127.12.126.67:45411)
0109 21:03:52.803407 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:52.803411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:52.803434 (+ 23us) server_negotiation.cc:407] Connection header received
0109 21:03:52.803525 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:52.803527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:52.803581 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:52.803695 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:52.804057 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.804571 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.805197 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.805396 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.805457 (+ 61us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:52.805921 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:52.806028 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:52.806117 (+ 89us) server_negotiation.cc:299] Negotiation successful
0109 21:03:52.806175 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
I20250109 21:03:52.809037 21169 raft_consensus.cc:1270] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff [term 1 FOLLOWER]: Refusing update from remote peer cb51b5220e604548a6da753363f95292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:52.809072 21029 raft_consensus.cc:1270] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92 [term 1 FOLLOWER]: Refusing update from remote peer cb51b5220e604548a6da753363f95292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:52.809549 21390 consensus_queue.cc:1035] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:52.809713 21459 consensus_queue.cc:1035] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:52.820246 20870 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58794:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:52.820402 20870 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58794:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:52.822391 20870 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967: Sending DeleteTablet for 3 replicas of tablet cf536da0ea4d4899966ca91671dfdae2
I20250109 21:03:52.823154 21009 tablet_service.cc:1514] Processing DeleteTablet for tablet cf536da0ea4d4899966ca91671dfdae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:52 UTC) from {username='oryx', principal='oryx/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:54426
I20250109 21:03:52.823161 21149 tablet_service.cc:1514] Processing DeleteTablet for tablet cf536da0ea4d4899966ca91671dfdae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:52 UTC) from {username='oryx', principal='oryx/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:37198
I20250109 21:03:52.823518 21473 tablet_replica.cc:331] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff: stopping tablet replica
I20250109 21:03:52.823630 21289 tablet_service.cc:1514] Processing DeleteTablet for tablet cf536da0ea4d4899966ca91671dfdae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:52 UTC) from {username='oryx', principal='oryx/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:52182
I20250109 21:03:52.823649 21473 raft_consensus.cc:2238] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:52.823784 21473 raft_consensus.cc:2267] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:52.823992 21474 tablet_replica.cc:331] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92: stopping tablet replica
I20250109 21:03:52.824265 21474 raft_consensus.cc:2238] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:52.824394 21474 raft_consensus.cc:2267] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:52.824883 20870 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 21:03:52.825145 21473 ts_tablet_manager.cc:1905] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 21:03:52.825350 20870 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:03:52.825419 21474 ts_tablet_manager.cc:1905] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:52.827390 21473 ts_tablet_manager.cc:1918] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:03:52.827468 21473 log.cc:1198] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/cf536da0ea4d4899966ca91671dfdae2
I20250109 21:03:52.827631 21474 ts_tablet_manager.cc:1918] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:03:52.827730 21474 log.cc:1198] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/wal/wals/cf536da0ea4d4899966ca91671dfdae2
I20250109 21:03:52.827832 21473 ts_tablet_manager.cc:1939] T cf536da0ea4d4899966ca91671dfdae2 P 4eec2002b5d9418c84bc049d10f686ff: Deleting consensus metadata
I20250109 21:03:52.828043 21474 ts_tablet_manager.cc:1939] T cf536da0ea4d4899966ca91671dfdae2 P 4976a64c48044865ab6c8917922e8f92: Deleting consensus metadata
I20250109 21:03:52.828691 21475 tablet_replica.cc:331] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292: stopping tablet replica
I20250109 21:03:52.828717 20856 catalog_manager.cc:4872] TS 4976a64c48044865ab6c8917922e8f92 (127.12.126.65:43869): tablet cf536da0ea4d4899966ca91671dfdae2 (table test-table [id=5838fdd8b5ad4fada785722c5d8ab060]) successfully deleted
I20250109 21:03:52.828822 21475 raft_consensus.cc:2238] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:03:52.828859 20856 catalog_manager.cc:4872] TS 4eec2002b5d9418c84bc049d10f686ff (127.12.126.66:33389): tablet cf536da0ea4d4899966ca91671dfdae2 (table test-table [id=5838fdd8b5ad4fada785722c5d8ab060]) successfully deleted
I20250109 21:03:52.829157 21475 raft_consensus.cc:2267] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:52.829885 21475 ts_tablet_manager.cc:1905] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:52.830906 21009 tablet_service.cc:1467] Processing CreateTablet for tablet b527915562164461a3797cfdfbeb8689 (DEFAULT_TABLE table=test-table [id=22499880b9994b84aa6be867956199cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:52.831175 21009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b527915562164461a3797cfdfbeb8689. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:52.831126 21289 tablet_service.cc:1467] Processing CreateTablet for tablet b527915562164461a3797cfdfbeb8689 (DEFAULT_TABLE table=test-table [id=22499880b9994b84aa6be867956199cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:52.831378 21289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b527915562164461a3797cfdfbeb8689. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:52.832770 21383 tablet_bootstrap.cc:492] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92: Bootstrap starting.
I20250109 21:03:52.832976 21149 tablet_service.cc:1467] Processing CreateTablet for tablet b527915562164461a3797cfdfbeb8689 (DEFAULT_TABLE table=test-table [id=22499880b9994b84aa6be867956199cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:03:52.833200 21149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b527915562164461a3797cfdfbeb8689. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:52.833523 21385 tablet_bootstrap.cc:492] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292: Bootstrap starting.
I20250109 21:03:52.833611 21383 tablet_bootstrap.cc:654] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:52.834442 21385 tablet_bootstrap.cc:654] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:52.835228 21384 tablet_bootstrap.cc:492] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff: Bootstrap starting.
I20250109 21:03:52.835803 21385 tablet_bootstrap.cc:492] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292: No bootstrap required, opened a new log
I20250109 21:03:52.835891 21385 ts_tablet_manager.cc:1397] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250109 21:03:52.836086 21384 tablet_bootstrap.cc:654] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:52.836341 21385 raft_consensus.cc:357] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.836459 21385 raft_consensus.cc:383] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:52.836498 21385 raft_consensus.cc:738] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb51b5220e604548a6da753363f95292, State: Initialized, Role: FOLLOWER
I20250109 21:03:52.836627 21385 consensus_queue.cc:260] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.836895 21385 ts_tablet_manager.cc:1428] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 21:03:52.837260 21384 tablet_bootstrap.cc:492] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff: No bootstrap required, opened a new log
I20250109 21:03:52.837338 21384 ts_tablet_manager.cc:1397] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250109 21:03:52.837787 21384 raft_consensus.cc:357] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.837913 21384 raft_consensus.cc:383] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:52.837955 21383 tablet_bootstrap.cc:492] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92: No bootstrap required, opened a new log
I20250109 21:03:52.837960 21384 raft_consensus.cc:738] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eec2002b5d9418c84bc049d10f686ff, State: Initialized, Role: FOLLOWER
I20250109 21:03:52.838053 21383 ts_tablet_manager.cc:1397] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20250109 21:03:52.838105 21384 consensus_queue.cc:260] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.838339 21384 ts_tablet_manager.cc:1428] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 21:03:52.838497 21383 raft_consensus.cc:357] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.838609 21383 raft_consensus.cc:383] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:52.838649 21383 raft_consensus.cc:738] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4976a64c48044865ab6c8917922e8f92, State: Initialized, Role: FOLLOWER
I20250109 21:03:52.838761 21383 consensus_queue.cc:260] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.839026 21383 ts_tablet_manager.cc:1428] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 21:03:52.841830 21475 ts_tablet_manager.cc:1918] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:03:52.841928 21475 log.cc:1198] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/cf536da0ea4d4899966ca91671dfdae2
I20250109 21:03:52.842231 21475 ts_tablet_manager.cc:1939] T cf536da0ea4d4899966ca91671dfdae2 P cb51b5220e604548a6da753363f95292: Deleting consensus metadata
I20250109 21:03:52.842876 20857 catalog_manager.cc:4872] TS cb51b5220e604548a6da753363f95292 (127.12.126.67:45411): tablet cf536da0ea4d4899966ca91671dfdae2 (table test-table [id=5838fdd8b5ad4fada785722c5d8ab060]) successfully deleted
I20250109 21:03:52.885253 21468 raft_consensus.cc:491] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:03:52.885459 21468 raft_consensus.cc:513] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.885955 21468 leader_election.cc:290] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4976a64c48044865ab6c8917922e8f92 (127.12.126.65:43869), 4eec2002b5d9418c84bc049d10f686ff (127.12.126.66:33389)
I20250109 21:03:52.886379 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b527915562164461a3797cfdfbeb8689" candidate_uuid: "cb51b5220e604548a6da753363f95292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4976a64c48044865ab6c8917922e8f92" is_pre_election: true
I20250109 21:03:52.886533 21029 raft_consensus.cc:2463] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb51b5220e604548a6da753363f95292 in term 0.
I20250109 21:03:52.886482 21169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b527915562164461a3797cfdfbeb8689" candidate_uuid: "cb51b5220e604548a6da753363f95292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eec2002b5d9418c84bc049d10f686ff" is_pre_election: true
I20250109 21:03:52.886634 21169 raft_consensus.cc:2463] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb51b5220e604548a6da753363f95292 in term 0.
I20250109 21:03:52.886863 21240 leader_election.cc:304] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4976a64c48044865ab6c8917922e8f92, cb51b5220e604548a6da753363f95292; no voters:
I20250109 21:03:52.887050 21468 raft_consensus.cc:2798] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:03:52.887104 21468 raft_consensus.cc:491] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:03:52.887151 21468 raft_consensus.cc:3054] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:52.888142 21468 raft_consensus.cc:513] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.888470 21468 leader_election.cc:290] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [CANDIDATE]: Term 1 election: Requested vote from peers 4976a64c48044865ab6c8917922e8f92 (127.12.126.65:43869), 4eec2002b5d9418c84bc049d10f686ff (127.12.126.66:33389)
I20250109 21:03:52.888871 21169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b527915562164461a3797cfdfbeb8689" candidate_uuid: "cb51b5220e604548a6da753363f95292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eec2002b5d9418c84bc049d10f686ff"
I20250109 21:03:52.888882 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b527915562164461a3797cfdfbeb8689" candidate_uuid: "cb51b5220e604548a6da753363f95292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4976a64c48044865ab6c8917922e8f92"
I20250109 21:03:52.888988 21169 raft_consensus.cc:3054] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:52.889007 21029 raft_consensus.cc:3054] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:52.889806 21169 raft_consensus.cc:2463] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb51b5220e604548a6da753363f95292 in term 1.
I20250109 21:03:52.890090 21240 leader_election.cc:304] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eec2002b5d9418c84bc049d10f686ff, cb51b5220e604548a6da753363f95292; no voters:
I20250109 21:03:52.890244 21029 raft_consensus.cc:2463] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb51b5220e604548a6da753363f95292 in term 1.
I20250109 21:03:52.890259 21468 raft_consensus.cc:2798] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:52.890352 21468 raft_consensus.cc:695] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [term 1 LEADER]: Becoming Leader. State: Replica: cb51b5220e604548a6da753363f95292, State: Running, Role: LEADER
I20250109 21:03:52.890506 21468 consensus_queue.cc:237] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } }
I20250109 21:03:52.892220 20871 catalog_manager.cc:5526] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb51b5220e604548a6da753363f95292 (127.12.126.67). New cstate: current_term: 1 leader_uuid: "cb51b5220e604548a6da753363f95292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb51b5220e604548a6da753363f95292" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45411 } health_report { overall_health: HEALTHY } } }
I20250109 21:03:52.917482 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.914527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:60011 (local address 127.12.126.65:43869)
0109 21:03:52.914718 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:52.914721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:52.914900 (+ 179us) server_negotiation.cc:407] Connection header received
0109 21:03:52.915040 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:52.915043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:52.915090 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:52.915154 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:52.915608 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.916068 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.916913 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.917044 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.917082 (+ 38us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:52.917170 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:52.917235 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:52.917327 (+ 92us) server_negotiation.cc:299] Negotiation successful
0109 21:03:52.917372 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20250109 21:03:52.917550 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.914378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:43869 (local address 127.12.126.67:60011)
0109 21:03:52.914787 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:52.914833 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:52.914945 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:52.915176 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:52.915179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:52.915186 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0109 21:03:52.915513 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:52.915521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.916160 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.916164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:52.916812 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:52.916822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.916930 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.916978 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0109 21:03:52.917243 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0109 21:03:52.917249 (+ 6us) client_negotiation.cc:769] Sending connection context
0109 21:03:52.917315 (+ 66us) client_negotiation.cc:240] Negotiation successful
0109 21:03:52.917387 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":142,"threads_started":1}
I20250109 21:03:52.923349 21169 raft_consensus.cc:1270] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff [term 1 FOLLOWER]: Refusing update from remote peer cb51b5220e604548a6da753363f95292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:52.923460 21029 raft_consensus.cc:1270] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92 [term 1 FOLLOWER]: Refusing update from remote peer cb51b5220e604548a6da753363f95292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:03:52.923753 21459 consensus_queue.cc:1035] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eec2002b5d9418c84bc049d10f686ff" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 33389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:52.923895 21468 consensus_queue.cc:1035] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4976a64c48044865ab6c8917922e8f92" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 43869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:03:52.944695 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.941604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:45411 (local address 127.12.126.65:42079)
0109 21:03:52.941888 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:52.941905 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:52.941998 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:52.942271 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:52.942274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:52.942288 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0109 21:03:52.942501 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:52.942508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.943425 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.943427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:52.943958 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:52.943971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.944064 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.944090 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0109 21:03:52.944450 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0109 21:03:52.944454 (+ 4us) client_negotiation.cc:769] Sending connection context
0109 21:03:52.944494 (+ 40us) client_negotiation.cc:240] Negotiation successful
0109 21:03:52.944562 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":98,"threads_started":1}
I20250109 21:03:52.944693 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.941931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:42079 (local address 127.12.126.67:45411)
0109 21:03:52.942051 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:52.942056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:52.942069 (+ 13us) server_negotiation.cc:407] Connection header received
0109 21:03:52.942111 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:52.942115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:52.942166 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:52.942256 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:03:52.942632 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.943320 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:52.944072 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:52.944244 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:52.944297 (+ 53us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:03:52.944371 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:03:52.944452 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:52.944518 (+ 66us) server_negotiation.cc:299] Negotiation successful
0109 21:03:52.944567 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250109 21:03:52.973173 20871 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58794:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:52.973363 20871 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58794:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:03:52.975683 20871 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ece6e8158f834a93afd6176c9d06b967: Sending DeleteTablet for 3 replicas of tablet b527915562164461a3797cfdfbeb8689
I20250109 21:03:52.976334 21289 tablet_service.cc:1514] Processing DeleteTablet for tablet b527915562164461a3797cfdfbeb8689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:52 UTC) from {username='oryx', principal='oryx/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:52182
I20250109 21:03:52.976361 21149 tablet_service.cc:1514] Processing DeleteTablet for tablet b527915562164461a3797cfdfbeb8689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:52 UTC) from {username='oryx', principal='oryx/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:37198
I20250109 21:03:52.976552 21475 tablet_replica.cc:331] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292: stopping tablet replica
I20250109 21:03:52.976578 21473 tablet_replica.cc:331] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff: stopping tablet replica
I20250109 21:03:52.976676 21475 raft_consensus.cc:2238] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:03:52.976699 21473 raft_consensus.cc:2238] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:52.976706 21009 tablet_service.cc:1514] Processing DeleteTablet for tablet b527915562164461a3797cfdfbeb8689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:03:52 UTC) from {username='oryx', principal='oryx/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:54426
I20250109 21:03:52.976824 21473 raft_consensus.cc:2267] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:52.976891 21475 raft_consensus.cc:2267] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:52.977114 21474 tablet_replica.cc:331] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92: stopping tablet replica
I20250109 21:03:52.977228 21474 raft_consensus.cc:2238] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:03:52.977317 21474 raft_consensus.cc:2267] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:03:52.978053 21475 ts_tablet_manager.cc:1905] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:52.978416 21473 ts_tablet_manager.cc:1905] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:52.978595 21474 ts_tablet_manager.cc:1905] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:03:52.980836 21474 ts_tablet_manager.cc:1918] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250109 21:03:52.980836 21473 ts_tablet_manager.cc:1918] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250109 21:03:52.980907 21474 log.cc:1198] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-0/wal/wals/b527915562164461a3797cfdfbeb8689
I20250109 21:03:52.980886 21475 ts_tablet_manager.cc:1918] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250109 21:03:52.980957 21475 log.cc:1198] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/b527915562164461a3797cfdfbeb8689
I20250109 21:03:52.981276 21475 ts_tablet_manager.cc:1939] T b527915562164461a3797cfdfbeb8689 P cb51b5220e604548a6da753363f95292: Deleting consensus metadata
I20250109 21:03:52.981277 21474 ts_tablet_manager.cc:1939] T b527915562164461a3797cfdfbeb8689 P 4976a64c48044865ab6c8917922e8f92: Deleting consensus metadata
I20250109 21:03:52.981612 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 20944
I20250109 21:03:52.983367 20857 catalog_manager.cc:4872] TS cb51b5220e604548a6da753363f95292 (127.12.126.67:45411): tablet b527915562164461a3797cfdfbeb8689 (table test-table [id=22499880b9994b84aa6be867956199cf]) successfully deleted
I20250109 21:03:52.986884 21473 log.cc:1198] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/b527915562164461a3797cfdfbeb8689
I20250109 21:03:52.987176 21473 ts_tablet_manager.cc:1939] T b527915562164461a3797cfdfbeb8689 P 4eec2002b5d9418c84bc049d10f686ff: Deleting consensus metadata
I20250109 21:03:52.987967 20856 catalog_manager.cc:4872] TS 4eec2002b5d9418c84bc049d10f686ff (127.12.126.66:33389): tablet b527915562164461a3797cfdfbeb8689 (table test-table [id=22499880b9994b84aa6be867956199cf]) successfully deleted
W20250109 21:03:52.991840 20856 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.126.65:43869 (error 108)
I20250109 21:03:52.992127 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 21084
I20250109 21:03:52.999338 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:52.992761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:43869 (local address 127.0.0.1:54462)
0109 21:03:52.999092 (+ 6331us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:52.999189 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.65:43869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6194,"thread_start_us":58,"threads_started":1}
W20250109 21:03:52.999485 20856 catalog_manager.cc:4615] TS 4976a64c48044865ab6c8917922e8f92 (127.12.126.65:43869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b527915562164461a3797cfdfbeb8689: Network error: Client connection negotiation failed: client connection to 127.12.126.65:43869: connect: Connection refused (error 111)
I20250109 21:03:53.000604 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 21224
I20250109 21:03:53.007524 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 20840
2025-01-09T21:03:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3358 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:03:53 dist-test-slave-g0zb krb5kdc[21495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:03:53 dist-test-slave-g0zb krb5kdc[21495](info): set up 2 sockets
Jan 09 21:03:53 dist-test-slave-g0zb krb5kdc[21495](info): commencing operation
krb5kdc: starting...
W20250109 21:03:54.028373 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:03:54 dist-test-slave-g0zb krb5kdc[21495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:03:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:03:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:03:54.143239 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:44987
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:45739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:44987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:54.250932 21508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:54.251169 21508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:54.251220 21508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:54.254292 21508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:54.254360 21508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:54.254383 21508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:54.254402 21508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:54.254419 21508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:03:54.258968 21508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:45739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:44987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:44987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:54.260063 21508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:54.260913 21508 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:54.268895 21513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:54.269742 21514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:54.270435 21516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:54.271596 21508 server_base.cc:1034] running on GCE node
I20250109 21:03:54.272150 21508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:54.273172 21508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:54.274333 21508 hybrid_clock.cc:648] HybridClock initialized: now 1736456634274310 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2025-01-09T21:03:54Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1272 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:03:54 dist-test-slave-g0zb krb5kdc[21521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:03:54 dist-test-slave-g0zb krb5kdc[21521](info): set up 2 sockets
Jan 09 21:03:54 dist-test-slave-g0zb krb5kdc[21521](info): commencing operation
krb5kdc: starting...
W20250109 21:03:55.300446 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:03:55 dist-test-slave-g0zb krb5kdc[21521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:03:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:03:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:03:55.418412 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:34575
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:34575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:55.525956 21534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:55.526209 21534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:55.526290 21534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:55.529419 21534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:55.529481 21534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:55.529503 21534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:55.529541 21534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:55.529557 21534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:03:55.533811 21534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:34575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:34575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:55.534905 21534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:55.535764 21534 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:55.543908 21539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:55.544727 21542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:55.545372 21540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:55.546346 21534 server_base.cc:1034] running on GCE node
I20250109 21:03:55.546947 21534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:55.547935 21534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:55.549116 21534 hybrid_clock.cc:648] HybridClock initialized: now 1736456635549102 us; error 32 us; skew 500 ppm
Jan 09 21:03:55 dist-test-slave-g0zb krb5kdc[21521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456635, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:55.551921 21534 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:03:55.552886 21534 webserver.cc:458] Webserver started at http://127.12.126.126:35373/ using document root <none> and password file <none>
I20250109 21:03:55.553465 21534 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:55.553525 21534 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:55.553709 21534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:55.555675 21534 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "43587a8c93224f9eb87a0a64215beb69"
format_stamp: "Formatted at 2025-01-09 21:03:55 on dist-test-slave-g0zb"
server_key: "ce3d0c9811a5141448cd3581d8842e08"
server_key_iv: "8785d00bc737f7f557621cf29ef51dfd"
server_key_version: "encryptionkey@0"
I20250109 21:03:55.556169 21534 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "43587a8c93224f9eb87a0a64215beb69"
format_stamp: "Formatted at 2025-01-09 21:03:55 on dist-test-slave-g0zb"
server_key: "ce3d0c9811a5141448cd3581d8842e08"
server_key_iv: "8785d00bc737f7f557621cf29ef51dfd"
server_key_version: "encryptionkey@0"
I20250109 21:03:55.559653 21534 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:03:55.561901 21549 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:55.562994 21534 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:03:55.563117 21534 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "43587a8c93224f9eb87a0a64215beb69"
format_stamp: "Formatted at 2025-01-09 21:03:55 on dist-test-slave-g0zb"
server_key: "ce3d0c9811a5141448cd3581d8842e08"
server_key_iv: "8785d00bc737f7f557621cf29ef51dfd"
server_key_version: "encryptionkey@0"
I20250109 21:03:55.563212 21534 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:55.580924 21534 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:55.584211 21534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:55.584419 21534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:55.591358 21534 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:34575
I20250109 21:03:55.591596 21601 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:34575 every 8 connection(s)
I20250109 21:03:55.592402 21534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:03:55.594775 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 21534
I20250109 21:03:55.594944 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:03:55.595072 21602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:03:55.595244 12793 external_mini_cluster.cc:1453] Setting key e41726b23b8f3e3e62e71fabf2ae0422
I20250109 21:03:55.603077 21602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69: Bootstrap starting.
Jan 09 21:03:55 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:55.605796 21602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69: Neither blocks nor log segments found. Creating new log.
I20250109 21:03:55.606662 21602 log.cc:826] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:55.608783 21602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69: No bootstrap required, opened a new log
I20250109 21:03:55.608850 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:55.596525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.12.126.126:34575)
0109 21:03:55.596934 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:55.596946 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:55.596989 (+ 43us) server_negotiation.cc:407] Connection header received
0109 21:03:55.597662 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:55.597688 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:55.598060 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:55.598482 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:55.599312 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:55.600461 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:55.601099 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:55.601481 (+ 382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:55.603974 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:55.604034 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:55.604048 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:55.604092 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:55.606208 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:55.606617 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:55.606623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:55.606628 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:55.606722 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:55.606996 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:55.607001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:55.607005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:55.607394 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:55.607595 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:55.607852 (+ 257us) server_negotiation.cc:299] Negotiation successful
0109 21:03:55.608111 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":121,"threads_started":1}
I20250109 21:03:55.612023 21602 raft_consensus.cc:357] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43587a8c93224f9eb87a0a64215beb69" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 34575 } }
I20250109 21:03:55.612215 21602 raft_consensus.cc:383] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:03:55.612267 21602 raft_consensus.cc:738] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43587a8c93224f9eb87a0a64215beb69, State: Initialized, Role: FOLLOWER
I20250109 21:03:55.612655 21602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43587a8c93224f9eb87a0a64215beb69" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 34575 } }
I20250109 21:03:55.612773 21602 raft_consensus.cc:397] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:03:55.612818 21602 raft_consensus.cc:491] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:03:55.612898 21602 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:03:55.613824 21602 raft_consensus.cc:513] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43587a8c93224f9eb87a0a64215beb69" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 34575 } }
I20250109 21:03:55.614126 21602 leader_election.cc:304] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43587a8c93224f9eb87a0a64215beb69; no voters:
I20250109 21:03:55.614365 21602 leader_election.cc:290] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:03:55.614475 21607 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:03:55.614706 21607 raft_consensus.cc:695] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [term 1 LEADER]: Becoming Leader. State: Replica: 43587a8c93224f9eb87a0a64215beb69, State: Running, Role: LEADER
I20250109 21:03:55.615056 21607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43587a8c93224f9eb87a0a64215beb69" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 34575 } }
I20250109 21:03:55.615479 21602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:55.616642 21609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43587a8c93224f9eb87a0a64215beb69. Latest consensus state: current_term: 1 leader_uuid: "43587a8c93224f9eb87a0a64215beb69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43587a8c93224f9eb87a0a64215beb69" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 34575 } } }
I20250109 21:03:55.616643 21608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43587a8c93224f9eb87a0a64215beb69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43587a8c93224f9eb87a0a64215beb69" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 34575 } } }
I20250109 21:03:55.616784 21608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:55.616784 21609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:55.617236 21616 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:03:55.620584 21616 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:03:55.626000 21616 catalog_manager.cc:1348] Generated new cluster ID: d55e40eb151742c699d613fb983655d1
I20250109 21:03:55.626080 21616 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:03:55.642295 21616 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:03:55.643327 21616 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:03:55.648917 21616 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69: Generated new TSK 0
I20250109 21:03:55.649621 21616 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
I20250109 21:03:55.686254 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:34575
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:55.795114 21628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:55.795387 21628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:55.795476 21628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:55.798619 21628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:55.798697 21628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:55.798843 21628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:03:55.803254 21628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:34575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:55.804400 21628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:55.805244 21628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:55.814142 21634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:55.815003 21636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:55.815771 21633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:55.816653 21628 server_base.cc:1034] running on GCE node
I20250109 21:03:55.817085 21628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:55.817739 21628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:55.818871 21628 hybrid_clock.cc:648] HybridClock initialized: now 1736456635818848 us; error 40 us; skew 500 ppm
Jan 09 21:03:55 dist-test-slave-g0zb krb5kdc[21521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456635, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:55.821576 21628 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:03:55.822502 21628 webserver.cc:458] Webserver started at http://127.12.126.65:37797/ using document root <none> and password file <none>
I20250109 21:03:55.823107 21628 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:55.823161 21628 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:55.823331 21628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:55.825323 21628 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "2fc9b71259484caa883fab79f0c7bffb"
format_stamp: "Formatted at 2025-01-09 21:03:55 on dist-test-slave-g0zb"
server_key: "368a47a068271fbb31d40763ee8fa9d5"
server_key_iv: "c7444b713bbbf57264ea03a6a3db3fe3"
server_key_version: "encryptionkey@0"
I20250109 21:03:55.825847 21628 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "2fc9b71259484caa883fab79f0c7bffb"
format_stamp: "Formatted at 2025-01-09 21:03:55 on dist-test-slave-g0zb"
server_key: "368a47a068271fbb31d40763ee8fa9d5"
server_key_iv: "c7444b713bbbf57264ea03a6a3db3fe3"
server_key_version: "encryptionkey@0"
I20250109 21:03:55.829409 21628 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250109 21:03:55.831645 21643 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:55.832666 21628 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:03:55.832782 21628 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "2fc9b71259484caa883fab79f0c7bffb"
format_stamp: "Formatted at 2025-01-09 21:03:55 on dist-test-slave-g0zb"
server_key: "368a47a068271fbb31d40763ee8fa9d5"
server_key_iv: "c7444b713bbbf57264ea03a6a3db3fe3"
server_key_version: "encryptionkey@0"
I20250109 21:03:55.832867 21628 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:55.848906 21628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:55.851703 21628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:55.851856 21628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:55.852375 21628 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:55.853188 21628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:55.853235 21628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:55.853272 21628 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:55.853287 21628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:55.861660 21628 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:39239
I20250109 21:03:55.861683 21756 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:39239 every 8 connection(s)
I20250109 21:03:55.862716 21628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
Jan 09 21:03:55 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456635, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:55.872802 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 21628
I20250109 21:03:55.872933 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:03:55.873199 12793 external_mini_cluster.cc:1453] Setting key 1ca06d8a420d35911bfe2d49c4a583ff
I20250109 21:03:55.875200 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:55.864561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:39095 (local address 127.12.126.126:34575)
0109 21:03:55.864691 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:55.864695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:55.865343 (+ 648us) server_negotiation.cc:407] Connection header received
0109 21:03:55.866577 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:55.866580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:55.866645 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:55.866720 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:55.868083 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:55.868592 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:55.869213 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:55.869391 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:55.871731 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:55.871752 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:55.871754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:55.871781 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:55.873145 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:55.873693 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:55.873696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:55.873699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:55.873775 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:55.874088 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:55.874091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:55.874093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:55.874269 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:55.874366 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:55.874932 (+ 566us) server_negotiation.cc:299] Negotiation successful
0109 21:03:55.875075 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250109 21:03:55.876291 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:55.864827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:39095)
0109 21:03:55.865209 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:55.865241 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:55.866361 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:55.866904 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:55.866913 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:55.867296 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:55.867916 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:55.867931 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:55.868711 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:55.868714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:55.869094 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:55.869101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:55.869338 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:55.869959 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:55.869983 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:55.871578 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:55.873262 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:55.873269 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:55.873283 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:55.873596 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:55.873874 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:55.873876 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:55.873878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:55.873992 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:55.874369 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:55.874373 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:55.874663 (+ 290us) client_negotiation.cc:769] Sending connection context
0109 21:03:55.874905 (+ 242us) client_negotiation.cc:240] Negotiation successful
0109 21:03:55.875347 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":89,"threads_started":1}
I20250109 21:03:55.877683 21757 heartbeater.cc:346] Connected to a master server at 127.12.126.126:34575
I20250109 21:03:55.877946 21757 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:55.878634 21757 heartbeater.cc:510] Master 127.12.126.126:34575 requested a full tablet report, sending...
I20250109 21:03:55.880434 21566 ts_manager.cc:194] Registered new tserver with Master: 2fc9b71259484caa883fab79f0c7bffb (127.12.126.65:39239)
I20250109 21:03:55.882126 21566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:39095
WARNING: no policy specified for kudu/127.12.126.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
I20250109 21:03:55.901976 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:34575
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:56.010838 21762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:56.011072 21762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:56.011161 21762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:56.014240 21762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:56.014317 21762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:56.014432 21762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:03:56.018769 21762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:34575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:56.019915 21762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:56.020781 21762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:56.029812 21767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:56.030472 21768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:56.031431 21770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:56.032285 21762 server_base.cc:1034] running on GCE node
I20250109 21:03:56.032712 21762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:56.033337 21762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:56.034498 21762 hybrid_clock.cc:648] HybridClock initialized: now 1736456636034471 us; error 44 us; skew 500 ppm
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456636, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:56.037384 21762 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:03:56.038357 21762 webserver.cc:458] Webserver started at http://127.12.126.66:46031/ using document root <none> and password file <none>
I20250109 21:03:56.038961 21762 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:56.039021 21762 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:56.039198 21762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:56.041002 21762 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "1aa453b6d35b400794748f276a26b781"
format_stamp: "Formatted at 2025-01-09 21:03:56 on dist-test-slave-g0zb"
server_key: "59d381a951ea81264fdd9aaf568efb43"
server_key_iv: "3e707a7e81b2207853bb9e5a9a011459"
server_key_version: "encryptionkey@0"
I20250109 21:03:56.041464 21762 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "1aa453b6d35b400794748f276a26b781"
format_stamp: "Formatted at 2025-01-09 21:03:56 on dist-test-slave-g0zb"
server_key: "59d381a951ea81264fdd9aaf568efb43"
server_key_iv: "3e707a7e81b2207853bb9e5a9a011459"
server_key_version: "encryptionkey@0"
I20250109 21:03:56.044850 21762 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:03:56.046984 21777 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:56.048033 21762 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:56.048131 21762 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "1aa453b6d35b400794748f276a26b781"
format_stamp: "Formatted at 2025-01-09 21:03:56 on dist-test-slave-g0zb"
server_key: "59d381a951ea81264fdd9aaf568efb43"
server_key_iv: "3e707a7e81b2207853bb9e5a9a011459"
server_key_version: "encryptionkey@0"
I20250109 21:03:56.048216 21762 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:56.060271 21762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:56.063308 21762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:56.063455 21762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:56.063917 21762 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:56.064659 21762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:56.064704 21762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:56.064743 21762 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:56.064764 21762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:56.073158 21762 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:43379
I20250109 21:03:56.073182 21890 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:43379 every 8 connection(s)
I20250109 21:03:56.074131 21762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:03:56.077862 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 21762
I20250109 21:03:56.078003 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:03:56.078323 12793 external_mini_cluster.cc:1453] Setting key 73f9ab837bc0ab0c65f7b0857ca4d169
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456636, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.126.67@KRBTEST.COM; defaulting to no policy
I20250109 21:03:56.086927 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:56.075891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:46819 (local address 127.12.126.126:34575)
0109 21:03:56.076042 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:56.076046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:56.076699 (+ 653us) server_negotiation.cc:407] Connection header received
0109 21:03:56.078103 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:56.078107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:56.078175 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:56.078258 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:56.079839 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.080321 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:56.081155 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.081287 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:56.083553 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:56.083571 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:56.083574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:56.083600 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:56.084997 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:56.085473 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:56.085477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:56.085479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:56.085524 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:56.085785 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:56.085788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:56.085790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:56.085938 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:56.086019 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:56.086566 (+ 547us) server_negotiation.cc:299] Negotiation successful
0109 21:03:56.086692 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20250109 21:03:56.087612 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:56.076165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:46819)
0109 21:03:56.076566 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:56.076602 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:56.077817 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:56.078464 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:56.078476 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:56.079017 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:56.079695 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:56.079712 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.080447 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:56.080453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:56.081052 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:56.081064 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.081296 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:56.081911 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:56.081935 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:56.083389 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:56.085116 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:56.085122 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:56.085135 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:56.085378 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:56.085604 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:56.085609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:56.085611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:56.085707 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:56.086031 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:56.086039 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:56.086314 (+ 275us) client_negotiation.cc:769] Sending connection context
0109 21:03:56.086536 (+ 222us) client_negotiation.cc:240] Negotiation successful
0109 21:03:56.086888 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":93,"threads_started":1}
I20250109 21:03:56.088708 21891 heartbeater.cc:346] Connected to a master server at 127.12.126.126:34575
I20250109 21:03:56.088954 21891 heartbeater.cc:463] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.67@KRBTEST.COM" created.
I20250109 21:03:56.089560 21891 heartbeater.cc:510] Master 127.12.126.126:34575 requested a full tablet report, sending...
I20250109 21:03:56.090737 21566 ts_manager.cc:194] Registered new tserver with Master: 1aa453b6d35b400794748f276a26b781 (127.12.126.66:43379)
I20250109 21:03:56.091357 21566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:46819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
I20250109 21:03:56.104003 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:34575
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:56.210134 21896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:56.210371 21896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:56.210431 21896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:56.213474 21896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:56.213539 21896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:56.213634 21896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:03:56.217999 21896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:34575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:56.219156 21896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:56.219997 21896 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:56.229040 21904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:56.229614 21902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:56.230574 21901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:56.231348 21896 server_base.cc:1034] running on GCE node
I20250109 21:03:56.231747 21896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:56.232348 21896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:56.233500 21896 hybrid_clock.cc:648] HybridClock initialized: now 1736456636233488 us; error 35 us; skew 500 ppm
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456636, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:56.236294 21896 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:03:56.237241 21896 webserver.cc:458] Webserver started at http://127.12.126.67:42033/ using document root <none> and password file <none>
I20250109 21:03:56.237783 21896 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:56.237834 21896 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:56.237993 21896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:03:56.239869 21896 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "7b7d728518a3407fbe6a0e4a8bc7a36b"
format_stamp: "Formatted at 2025-01-09 21:03:56 on dist-test-slave-g0zb"
server_key: "1272c92883a0e46a2d29bc5ad2ed2f17"
server_key_iv: "d2f7558b36f907a583645600a3ac20e3"
server_key_version: "encryptionkey@0"
I20250109 21:03:56.240309 21896 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "7b7d728518a3407fbe6a0e4a8bc7a36b"
format_stamp: "Formatted at 2025-01-09 21:03:56 on dist-test-slave-g0zb"
server_key: "1272c92883a0e46a2d29bc5ad2ed2f17"
server_key_iv: "d2f7558b36f907a583645600a3ac20e3"
server_key_version: "encryptionkey@0"
I20250109 21:03:56.243703 21896 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:03:56.245987 21911 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:56.247058 21896 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:03:56.247154 21896 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "7b7d728518a3407fbe6a0e4a8bc7a36b"
format_stamp: "Formatted at 2025-01-09 21:03:56 on dist-test-slave-g0zb"
server_key: "1272c92883a0e46a2d29bc5ad2ed2f17"
server_key_iv: "d2f7558b36f907a583645600a3ac20e3"
server_key_version: "encryptionkey@0"
I20250109 21:03:56.247236 21896 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:56.264904 21896 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:56.268069 21896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:56.268225 21896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:56.268731 21896 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:56.269547 21896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:56.269593 21896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:56.269644 21896 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:56.269665 21896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:56.277966 21896 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:33367
I20250109 21:03:56.277998 22024 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:33367 every 8 connection(s)
I20250109 21:03:56.279027 21896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:03:56.279690 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 21896
I20250109 21:03:56.279816 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:03:56.280069 12793 external_mini_cluster.cc:1453] Setting key 3858e302a98ace4007039670f8c7053d
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456636, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:03:56.291391 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:56.280854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:42471 (local address 127.12.126.126:34575)
0109 21:03:56.280987 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:56.280990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:56.281781 (+ 791us) server_negotiation.cc:407] Connection header received
0109 21:03:56.283149 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:56.283152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:56.283211 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:56.283299 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:56.284638 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.285134 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:56.285759 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.285930 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:56.288267 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:56.288286 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:56.288288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:56.288316 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:56.289535 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:56.289982 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:56.289985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:56.289987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:56.290031 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:56.290290 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:56.290293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:56.290295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:56.290445 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:56.290533 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:56.291058 (+ 525us) server_negotiation.cc:299] Negotiation successful
0109 21:03:56.291191 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250109 21:03:56.292016 22027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:56.281193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.67:42471)
0109 21:03:56.281660 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:56.281702 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:56.282943 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:56.283493 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:56.283502 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:56.283918 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:56.284499 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:56.284513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.285249 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:56.285252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:56.285642 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:56.285649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.285899 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:56.286638 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:56.286661 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:56.288112 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:03:56.289646 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:56.289651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:56.289663 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:56.289891 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:56.290109 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:03:56.290112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:03:56.290114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:03:56.290194 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:03:56.290545 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:03:56.290553 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:03:56.290781 (+ 228us) client_negotiation.cc:769] Sending connection context
0109 21:03:56.291003 (+ 222us) client_negotiation.cc:240] Negotiation successful
0109 21:03:56.291302 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":102,"threads_started":1}
I20250109 21:03:56.293123 22025 heartbeater.cc:346] Connected to a master server at 127.12.126.126:34575
I20250109 21:03:56.293397 22025 heartbeater.cc:463] Registering TS with master...
I20250109 21:03:56.294041 22025 heartbeater.cc:510] Master 127.12.126.126:34575 requested a full tablet report, sending...
I20250109 21:03:56.295038 21566 ts_manager.cc:194] Registered new tserver with Master: 7b7d728518a3407fbe6a0e4a8bc7a36b (127.12.126.67:33367)
I20250109 21:03:56.295578 21566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:42471
I20250109 21:03:56.305928 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.65.keytab.
Entry for principal oryx/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.65.keytab.
WARNING: no policy specified for oryx/127.12.126.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.126.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.66.keytab.
Entry for principal oryx/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.66.keytab.
WARNING: no policy specified for oryx/127.12.126.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.126.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.67.keytab.
Entry for principal oryx/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.67.keytab.
I20250109 21:03:56.375857 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 21628
I20250109 21:03:56.381805 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 21762
I20250109 21:03:56.387413 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 21896
I20250109 21:03:56.392903 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 21534
I20250109 21:03:56.399662 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:34575
--webserver_interface=127.12.126.126
--webserver_port=35373
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:34575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:03:56.507071 22034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:56.507361 22034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:56.507442 22034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:56.510731 22034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:03:56.510850 22034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:56.510902 22034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:56.510942 22034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:03:56.510977 22034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:03:56.515643 22034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:34575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:34575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=35373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:56.516831 22034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:56.517736 22034 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:56.525756 22039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:56.526675 22040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:56.527395 22042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:56.528375 22034 server_base.cc:1034] running on GCE node
I20250109 21:03:56.528936 22034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:56.529922 22034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:56.531093 22034 hybrid_clock.cc:648] HybridClock initialized: now 1736456636531082 us; error 31 us; skew 500 ppm
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456636, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:56.533793 22034 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:03:56.534766 22034 webserver.cc:458] Webserver started at http://127.12.126.126:35373/ using document root <none> and password file <none>
I20250109 21:03:56.535360 22034 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:56.535429 22034 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:56.538952 22034 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20250109 21:03:56.540879 22049 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:56.541899 22034 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:56.541996 22034 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "43587a8c93224f9eb87a0a64215beb69"
format_stamp: "Formatted at 2025-01-09 21:03:55 on dist-test-slave-g0zb"
server_key: "ce3d0c9811a5141448cd3581d8842e08"
server_key_iv: "8785d00bc737f7f557621cf29ef51dfd"
server_key_version: "encryptionkey@0"
I20250109 21:03:56.542475 22034 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:56.554885 22034 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:56.558029 22034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:56.558190 22034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:56.566985 22034 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:34575
I20250109 21:03:56.567209 22101 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:34575 every 8 connection(s)
I20250109 21:03:56.568162 22034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:03:56.572211 22102 sys_catalog.cc:263] Verifying existing consensus state
I20250109 21:03:56.573798 22102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69: Bootstrap starting.
I20250109 21:03:56.575408 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 22034
I20250109 21:03:56.575937 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:39239
--local_ip_for_outbound_sockets=127.12.126.65
--tserver_master_addrs=127.12.126.126:34575
--webserver_port=37797
--webserver_interface=127.12.126.65
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.126.65
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:03:56.590413 22102 log.cc:826] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69: Log is configured to *not* fsync() on all Append() calls
I20250109 21:03:56.596613 22102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:03:56.596982 22102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69: Bootstrap complete.
I20250109 21:03:56.600615 22102 raft_consensus.cc:357] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43587a8c93224f9eb87a0a64215beb69" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 34575 } }
I20250109 21:03:56.600929 22102 raft_consensus.cc:738] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43587a8c93224f9eb87a0a64215beb69, State: Initialized, Role: FOLLOWER
I20250109 21:03:56.601416 22102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43587a8c93224f9eb87a0a64215beb69" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 34575 } }
I20250109 21:03:56.601536 22102 raft_consensus.cc:397] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:03:56.601599 22102 raft_consensus.cc:491] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:03:56.601696 22102 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:03:56.603418 22102 raft_consensus.cc:513] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43587a8c93224f9eb87a0a64215beb69" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 34575 } }
I20250109 21:03:56.603739 22102 leader_election.cc:304] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43587a8c93224f9eb87a0a64215beb69; no voters:
I20250109 21:03:56.604085 22102 leader_election.cc:290] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250109 21:03:56.604173 22106 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [term 2 FOLLOWER]: Leader election won for term 2
I20250109 21:03:56.604409 22106 raft_consensus.cc:695] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [term 2 LEADER]: Becoming Leader. State: Replica: 43587a8c93224f9eb87a0a64215beb69, State: Running, Role: LEADER
I20250109 21:03:56.604781 22106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43587a8c93224f9eb87a0a64215beb69" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 34575 } }
I20250109 21:03:56.605324 22102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:03:56.606348 22108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43587a8c93224f9eb87a0a64215beb69. Latest consensus state: current_term: 2 leader_uuid: "43587a8c93224f9eb87a0a64215beb69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43587a8c93224f9eb87a0a64215beb69" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 34575 } } }
I20250109 21:03:56.606511 22108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:56.606698 22107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "43587a8c93224f9eb87a0a64215beb69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43587a8c93224f9eb87a0a64215beb69" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 34575 } } }
I20250109 21:03:56.606781 22107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69 [sys.catalog]: This master's current role is: LEADER
I20250109 21:03:56.610895 22122 catalog_manager.cc:1260] Loaded cluster ID: d55e40eb151742c699d613fb983655d1
I20250109 21:03:56.610963 22122 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69: loading cluster ID for follower catalog manager: success
I20250109 21:03:56.613559 22122 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69: acquiring CA information for follower catalog manager: success
I20250109 21:03:56.615054 22122 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 21:03:56.615187 22115 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:03:56.615914 22115 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:03:56.616282 22115 catalog_manager.cc:1260] Loaded cluster ID: d55e40eb151742c699d613fb983655d1
I20250109 21:03:56.616338 22115 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:03:56.616809 22115 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:03:56.617167 22115 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 43587a8c93224f9eb87a0a64215beb69: Loaded TSK: 0
I20250109 21:03:56.617727 22115 catalog_manager.cc:1515] Initializing in-progress tserver states...
W20250109 21:03:56.687397 22104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:56.687644 22104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:56.687709 22104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:56.690898 22104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:56.690963 22104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:56.691067 22104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:03:56.695292 22104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.65.keytab
--principal=oryx/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:39239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=37797
--tserver_master_addrs=127.12.126.126:34575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:56.696421 22104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:56.697285 22104 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:56.706473 22129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:56.707129 22131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:56.707993 22128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:56.708914 22104 server_base.cc:1034] running on GCE node
I20250109 21:03:56.709347 22104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:56.709964 22104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:56.711125 22104 hybrid_clock.cc:648] HybridClock initialized: now 1736456636711099 us; error 45 us; skew 500 ppm
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456636, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:56.713907 22104 init.cc:377] Logged in from keytab as oryx/127.12.126.65@KRBTEST.COM (short username oryx)
I20250109 21:03:56.714979 22104 webserver.cc:458] Webserver started at http://127.12.126.65:37797/ using document root <none> and password file <none>
I20250109 21:03:56.715548 22104 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:56.715617 22104 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:56.719228 22104 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20250109 21:03:56.721169 22138 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:56.722249 22104 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:56.722368 22104 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "2fc9b71259484caa883fab79f0c7bffb"
format_stamp: "Formatted at 2025-01-09 21:03:55 on dist-test-slave-g0zb"
server_key: "368a47a068271fbb31d40763ee8fa9d5"
server_key_iv: "c7444b713bbbf57264ea03a6a3db3fe3"
server_key_version: "encryptionkey@0"
I20250109 21:03:56.722833 22104 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:56.736419 22104 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:56.739554 22104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:56.739733 22104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:56.740270 22104 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:56.741056 22104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:56.741106 22104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:56.741164 22104 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:56.741190 22104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:56.751543 22104 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:39239
I20250109 21:03:56.751580 22251 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:39239 every 8 connection(s)
I20250109 21:03:56.752697 22104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:03:56.762171 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 22104
I20250109 21:03:56.762687 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:43379
--local_ip_for_outbound_sockets=127.12.126.66
--tserver_master_addrs=127.12.126.126:34575
--webserver_port=46031
--webserver_interface=127.12.126.66
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.126.66
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
W20250109 21:03:56.765875 22254 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:56.754869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:36147 (local address 127.12.126.126:34575)
0109 21:03:56.755330 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:56.755343 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:56.755790 (+ 447us) server_negotiation.cc:407] Connection header received
0109 21:03:56.757457 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:56.757468 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:56.757722 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:56.757994 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:56.759627 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.760418 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:56.761198 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.761365 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:56.765315 (+ 3950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.65:36147: BlockingRecv error: recv got EOF from 127.12.126.65:36147 (error 108)
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":113,"threads_started":1}
I20250109 21:03:56.765961 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:56.754876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:36147)
0109 21:03:56.755626 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:56.755668 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:56.757067 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:56.758140 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:56.758150 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:56.758701 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:56.759464 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:56.759482 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.760551 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:56.760555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:56.761094 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:56.761105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.761321 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:56.761994 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:56.762027 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:56.765110 (+ 3083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":115,"threads_started":1}
W20250109 21:03:56.766613 22252 heartbeater.cc:643] Failed to heartbeat to 127.12.126.126:34575 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.126.126:34575: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:03:56.772733 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:56.767072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:55017)
0109 21:03:56.767318 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:56.767333 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:56.767417 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:56.767684 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:56.767687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:56.767853 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:56.768135 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:56.768142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.769185 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:56.769189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:56.769778 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:56.769787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.769893 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:56.770413 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:56.770427 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:56.772580 (+ 2153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163}
W20250109 21:03:56.772939 22254 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:56.767172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:55017 (local address 127.12.126.126:34575)
0109 21:03:56.767347 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:56.767353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:56.767376 (+ 23us) server_negotiation.cc:407] Connection header received
0109 21:03:56.767509 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:56.767515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:56.767574 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:56.767665 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:56.768251 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.769070 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:56.769897 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.770119 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:56.772782 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.65:55017: BlockingRecv error: recv got EOF from 127.12.126.65:55017 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:03:56.778841 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:56.773516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:40415)
0109 21:03:56.773796 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:56.773811 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:56.773886 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:56.774187 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:56.774192 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:56.774346 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:56.774545 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:56.774553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.775594 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:56.775598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:56.776061 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:56.776070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.776177 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:56.776692 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:56.776707 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:56.778654 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206}
W20250109 21:03:56.778988 22254 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:56.773783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:40415 (local address 127.12.126.126:34575)
0109 21:03:56.773922 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:56.773927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:56.773942 (+ 15us) server_negotiation.cc:407] Connection header received
0109 21:03:56.773998 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:56.774003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:56.774059 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:56.774158 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:56.774668 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.775472 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:56.776183 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.776405 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:56.778842 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.65:40415: BlockingRecv error: recv got EOF from 127.12.126.65:40415 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20250109 21:03:56.779076 22252 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250109 21:03:56.876788 22256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:56.877063 22256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:56.877156 22256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:56.880350 22256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:56.880425 22256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:56.880544 22256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:03:56.884972 22256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.66.keytab
--principal=oryx/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:43379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=46031
--tserver_master_addrs=127.12.126.126:34575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:56.886153 22256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:56.887042 22256 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:56.895818 22262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:56.896620 22261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:56.897429 22264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:56.898365 22256 server_base.cc:1034] running on GCE node
I20250109 21:03:56.898838 22256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:56.899480 22256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:56.900637 22256 hybrid_clock.cc:648] HybridClock initialized: now 1736456636900629 us; error 28 us; skew 500 ppm
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456636, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:56.903553 22256 init.cc:377] Logged in from keytab as oryx/127.12.126.66@KRBTEST.COM (short username oryx)
I20250109 21:03:56.904506 22256 webserver.cc:458] Webserver started at http://127.12.126.66:46031/ using document root <none> and password file <none>
I20250109 21:03:56.905059 22256 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:56.905133 22256 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:56.908672 22256 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20250109 21:03:56.910480 22271 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:56.911490 22256 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:56.911631 22256 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "1aa453b6d35b400794748f276a26b781"
format_stamp: "Formatted at 2025-01-09 21:03:56 on dist-test-slave-g0zb"
server_key: "59d381a951ea81264fdd9aaf568efb43"
server_key_iv: "3e707a7e81b2207853bb9e5a9a011459"
server_key_version: "encryptionkey@0"
I20250109 21:03:56.912040 22256 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:56.946166 22256 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:56.949092 22256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:56.949277 22256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:56.949836 22256 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:56.950640 22256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:56.950690 22256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:56.950752 22256 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:56.950779 22256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:56.959609 22256 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:43379
I20250109 21:03:56.959643 22384 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:43379 every 8 connection(s)
I20250109 21:03:56.960675 22256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:03:56.961987 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 22256
I20250109 21:03:56.962479 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:33367
--local_ip_for_outbound_sockets=127.12.126.67
--tserver_master_addrs=127.12.126.126:34575
--webserver_port=42033
--webserver_interface=127.12.126.67
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.126.67
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
W20250109 21:03:56.971123 22254 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:56.962501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:55611 (local address 127.12.126.126:34575)
0109 21:03:56.962697 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:56.962703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:56.963561 (+ 858us) server_negotiation.cc:407] Connection header received
0109 21:03:56.965056 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:56.965059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:56.965117 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:56.965214 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:56.966657 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.967178 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:56.967817 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.967989 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:56.970998 (+ 3009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.66:55611: BlockingRecv error: recv got EOF from 127.12.126.66:55611 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20250109 21:03:56.971601 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:56.962874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:55611)
0109 21:03:56.963375 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:56.963437 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:56.964818 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:56.965483 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:56.965491 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:56.965930 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:56.966504 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:56.966518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.967318 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:56.967320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:56.967707 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:56.967714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.967936 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:56.968694 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:56.968717 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:56.970856 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":98,"threads_started":1}
W20250109 21:03:56.972039 22385 heartbeater.cc:643] Failed to heartbeat to 127.12.126.126:34575 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.126.126:34575: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:03:56.978106 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:56.972481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:54713)
0109 21:03:56.972595 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:56.972613 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:56.972678 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:56.972928 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:56.972930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:56.973053 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:56.973225 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:56.973230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.974248 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:56.974252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:56.974885 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:56.974896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.975003 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:56.975883 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:56.975901 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:56.977965 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20250109 21:03:56.978451 22254 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:56.972547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:54713 (local address 127.12.126.126:34575)
0109 21:03:56.972702 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:56.972707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:56.972723 (+ 16us) server_negotiation.cc:407] Connection header received
0109 21:03:56.972766 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:56.972772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:56.972825 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:56.972922 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:56.973337 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.974107 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:56.975036 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.975295 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:56.978109 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.66:54713: BlockingRecv error: recv got EOF from 127.12.126.66:54713 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:56 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:03:56.983040 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:56.978623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:48109)
0109 21:03:56.978742 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:56.978755 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:56.978858 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:56.979234 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:56.979236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:56.979376 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:56.979528 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:56.979534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.980409 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:56.980411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:56.980749 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:56.980754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.980832 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:56.981304 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:56.981316 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:56.982918 (+ 1602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20250109 21:03:56.983233 22385 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250109 21:03:56.983131 22254 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:56.978900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:48109 (local address 127.12.126.126:34575)
0109 21:03:56.979026 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:56.979031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:56.979044 (+ 13us) server_negotiation.cc:407] Connection header received
0109 21:03:56.979082 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:56.979084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:56.979129 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:56.979211 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:56.979628 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.980319 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:56.980848 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:56.981046 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:56.983032 (+ 1986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.66:48109: BlockingRecv error: recv got EOF from 127.12.126.66:48109 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20250109 21:03:57.074842 22388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:03:57.075191 22388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:03:57.075336 22388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:03:57.078527 22388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:03:57.078606 22388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:03:57.078724 22388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:03:57.083084 22388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/krb5kdc/oryx_127.12.126.67.keytab
--principal=oryx/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:33367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=42033
--tserver_master_addrs=127.12.126.126:34575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:03:57.084228 22388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:03:57.085112 22388 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:03:57.093904 22396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:57.094568 22393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:03:57.095618 22394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:03:57.096495 22388 server_base.cc:1034] running on GCE node
I20250109 21:03:57.096940 22388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:03:57.097546 22388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:03:57.098702 22388 hybrid_clock.cc:648] HybridClock initialized: now 1736456637098691 us; error 28 us; skew 500 ppm
Jan 09 21:03:57 dist-test-slave-g0zb krb5kdc[21521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456637, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:03:57.101583 22388 init.cc:377] Logged in from keytab as oryx/127.12.126.67@KRBTEST.COM (short username oryx)
I20250109 21:03:57.102597 22388 webserver.cc:458] Webserver started at http://127.12.126.67:42033/ using document root <none> and password file <none>
I20250109 21:03:57.103211 22388 fs_manager.cc:362] Metadata directory not provided
I20250109 21:03:57.103294 22388 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:03:57.107088 22388 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20250109 21:03:57.109136 22403 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:57.110168 22388 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:03:57.110281 22388 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "7b7d728518a3407fbe6a0e4a8bc7a36b"
format_stamp: "Formatted at 2025-01-09 21:03:56 on dist-test-slave-g0zb"
server_key: "1272c92883a0e46a2d29bc5ad2ed2f17"
server_key_iv: "d2f7558b36f907a583645600a3ac20e3"
server_key_version: "encryptionkey@0"
I20250109 21:03:57.110724 22388 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:03:57.128443 22388 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:03:57.131186 22388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:03:57.131357 22388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:03:57.131876 22388 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:03:57.132651 22388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:03:57.132699 22388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:57.132746 22388 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:03:57.132776 22388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:03:57.141395 22388 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:33367
I20250109 21:03:57.141424 22516 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:33367 every 8 connection(s)
I20250109 21:03:57.142489 22388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:03:57.149123 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 22388
Jan 09 21:03:57 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:57 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
W20250109 21:03:57.153770 22254 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:57.144230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:56437 (local address 127.12.126.126:34575)
0109 21:03:57.144375 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:57.144379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:57.145040 (+ 661us) server_negotiation.cc:407] Connection header received
0109 21:03:57.146220 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:57.146223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:57.146275 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:57.146376 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:57.147741 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.148588 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:57.149224 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.149549 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:57.153596 (+ 4047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.67:56437: BlockingRecv error: recv got EOF from 127.12.126.67:56437 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20250109 21:03:57.154235 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:57.144502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.67:56437)
0109 21:03:57.144905 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:57.144938 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:57.146028 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:57.146499 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:57.146508 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:57.146952 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:57.147563 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:57.147578 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.148709 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:57.148712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:57.149103 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:57.149110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.149359 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:57.150090 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:57.150115 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:57.153407 (+ 3292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":92,"threads_started":1}
W20250109 21:03:57.154753 22517 heartbeater.cc:643] Failed to heartbeat to 127.12.126.126:34575 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.126.126:34575: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Jan 09 21:03:57 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:57 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:03:57.161885 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:57.155259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.67:54653)
0109 21:03:57.155418 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:57.155435 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:57.155526 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:57.156719 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:57.156722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:57.156899 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:57.157161 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:57.157173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.158234 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:57.158239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:57.158827 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:57.158840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.158958 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:57.159441 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:57.159476 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:57.161742 (+ 2266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20250109 21:03:57.162343 22254 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:57.156333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:54653 (local address 127.12.126.126:34575)
0109 21:03:57.156477 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:57.156481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:57.156494 (+ 13us) server_negotiation.cc:407] Connection header received
0109 21:03:57.156540 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:57.156544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:57.156597 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:57.156686 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:57.157285 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.157734 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:57.158973 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.159263 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:57.161910 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.67:54653: BlockingRecv error: recv got EOF from 127.12.126.67:54653 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20250109 21:03:57.165028 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:57.151190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59052 (local address 127.12.126.126:34575)
0109 21:03:57.151527 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:57.151531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:57.151753 (+ 222us) server_negotiation.cc:407] Connection header received
0109 21:03:57.151897 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:57.151901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:57.151978 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:57.152084 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:57.153342 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.154549 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:57.155809 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.156017 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:57.157358 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:03:57.157383 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:03:57.157402 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:03:57.157457 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:03:57.160089 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:57.160819 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:57.160827 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:57.160834 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:57.160917 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:03:57.163076 (+ 2159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:03:57.163079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:03:57.163081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:03:57.163549 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:03:57.163753 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0109 21:03:57.164716 (+ 963us) server_negotiation.cc:299] Negotiation successful
0109 21:03:57.164910 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":98,"threads_started":1}
Jan 09 21:03:57 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:57 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:03:57.168769 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:57.162489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.67:54825)
0109 21:03:57.162741 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:57.162756 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:57.162860 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:57.163337 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:57.163342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:57.163525 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:57.163758 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:57.163767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.164588 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:57.164593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:57.165187 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:57.165200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.165333 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:57.165879 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:57.165896 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:57.168521 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173}
W20250109 21:03:57.168771 22254 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:57.162724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:54825 (local address 127.12.126.126:34575)
0109 21:03:57.162943 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:57.162949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:57.162965 (+ 16us) server_negotiation.cc:407] Connection header received
0109 21:03:57.163005 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:57.163010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:57.163067 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:57.163158 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:57.163866 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.164495 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:57.165311 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.165439 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:57.168656 (+ 3217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.67:54825: BlockingRecv error: recv got EOF from 127.12.126.67:54825 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20250109 21:03:57.168963 22517 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jan 09 21:03:57 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:57 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:03:57.790055 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:57.779659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:38901)
0109 21:03:57.779927 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:57.779940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:57.780036 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:57.780321 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:57.780324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:57.780501 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:57.780737 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:57.780743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.781425 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:57.781427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:57.781796 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:57.781802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.781882 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:57.787976 (+ 6094us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:57.787994 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:57.789881 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
W20250109 21:03:57.790143 22523 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:57.779726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:38901 (local address 127.12.126.126:34575)
0109 21:03:57.779957 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:57.779961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:57.779987 (+ 26us) server_negotiation.cc:407] Connection header received
0109 21:03:57.780148 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:57.780151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:57.780199 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:57.780287 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:57.780829 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.781343 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:57.781869 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.781984 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:57.790003 (+ 8019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.65:38901: BlockingRecv error: recv got EOF from 127.12.126.65:38901 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":77,"threads_started":1}
Jan 09 21:03:57 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:57 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:03:57.988395 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:57.983856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:51087)
0109 21:03:57.984112 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:57.984124 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:57.984225 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:57.984453 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:57.984455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:57.984625 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:57.984842 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:57.984848 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.985495 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:57.985497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:57.985862 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:57.985868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.985954 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:57.986494 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:57.986508 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:57.988245 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":75,"threads_started":1}
W20250109 21:03:57.988494 22523 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:57.983915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:51087 (local address 127.12.126.126:34575)
0109 21:03:57.984046 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:57.984050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:57.984185 (+ 135us) server_negotiation.cc:407] Connection header received
0109 21:03:57.984310 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:57.984312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:57.984361 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:57.984431 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:57.984930 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.985417 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:57.985938 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:57.986054 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:57.988391 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.66:51087: BlockingRecv error: recv got EOF from 127.12.126.66:51087 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jan 09 21:03:58 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:58 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:03:58.174315 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:58.169555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.67:43071)
0109 21:03:58.169861 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:58.169874 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:58.169988 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:58.170230 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:58.170233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:58.170435 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:58.170660 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:58.170667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:58.171420 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:58.171424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:58.171798 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:58.171805 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:58.171913 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:58.172457 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:58.172469 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:58.174131 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":102,"threads_started":1}
W20250109 21:03:58.174373 22523 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:58.169647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:43071 (local address 127.12.126.126:34575)
0109 21:03:58.169795 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:58.169812 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:58.169943 (+ 131us) server_negotiation.cc:407] Connection header received
0109 21:03:58.170080 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:58.170083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:58.170127 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:58.170198 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:58.170780 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:58.171311 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:58.171906 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:58.172107 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:58.174259 (+ 2152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.67:43071: BlockingRecv error: recv got EOF from 127.12.126.67:43071 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jan 09 21:03:58 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:58 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:03:58.795516 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:58.791070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:43471)
0109 21:03:58.791366 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:58.791380 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:58.791474 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:58.791700 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:58.791702 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:58.791880 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:58.792078 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:58.792084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:58.792765 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:58.792768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:58.793128 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:58.793135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:58.793224 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:58.793735 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:58.793746 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:58.795381 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":92,"threads_started":1}
W20250109 21:03:58.795658 22527 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:58.791170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:43471 (local address 127.12.126.126:34575)
0109 21:03:58.791456 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:58.791459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:58.791473 (+ 14us) server_negotiation.cc:407] Connection header received
0109 21:03:58.791552 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:58.791554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:58.791600 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:58.791676 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:58.792165 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:58.792682 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:58.793212 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:58.793327 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:58.795544 (+ 2217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.65:43471: BlockingRecv error: recv got EOF from 127.12.126.65:43471 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":109,"threads_started":1}
Jan 09 21:03:58 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:58 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:03:58.993841 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:58.989137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:47651)
0109 21:03:58.989395 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:58.989408 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:58.989525 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:58.989758 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:58.989760 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:58.989938 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:58.990131 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:58.990136 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:58.990871 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:58.990876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:58.991265 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:58.991272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:58.991365 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:58.991899 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:58.991911 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:58.993722 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
W20250109 21:03:58.993996 22527 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:58.989206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:47651 (local address 127.12.126.126:34575)
0109 21:03:58.989368 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:58.989372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:58.989518 (+ 146us) server_negotiation.cc:407] Connection header received
0109 21:03:58.989611 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:58.989613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:58.989664 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:58.989738 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:58.990221 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:58.990710 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:58.991358 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:58.991501 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:58.993862 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.66:47651: BlockingRecv error: recv got EOF from 127.12.126.66:47651 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jan 09 21:03:59 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:59 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:03:59.179755 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:59.175176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.67:38761)
0109 21:03:59.175455 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:59.175471 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:59.175600 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:59.175873 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:59.175877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:59.176088 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:59.176289 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:59.176295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:59.176948 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:59.176950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:59.177301 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:59.177306 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:59.177388 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:59.177915 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:59.177926 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:59.179639 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":91,"threads_started":1}
W20250109 21:03:59.179924 22527 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:59.175346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:38761 (local address 127.12.126.126:34575)
0109 21:03:59.175497 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:59.175503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:59.175537 (+ 34us) server_negotiation.cc:407] Connection header received
0109 21:03:59.175697 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:59.175700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:59.175762 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:59.175846 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:59.176387 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:59.176865 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:59.177377 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:59.177488 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:59.179777 (+ 2289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.67:38761: BlockingRecv error: recv got EOF from 127.12.126.67:38761 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jan 09 21:03:59 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:59 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:03:59.801326 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:59.796376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:38697)
0109 21:03:59.796868 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:59.796884 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:59.796999 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:59.797256 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:59.797258 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:59.797433 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:59.797689 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:59.797697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:59.798515 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:59.798518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:59.798910 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:59.798917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:59.799007 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:59.799578 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:59.799590 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:59.801205 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":111,"threads_started":1}
W20250109 21:03:59.801492 22531 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:59.796554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:38697 (local address 127.12.126.126:34575)
0109 21:03:59.796819 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:59.796823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:59.796958 (+ 135us) server_negotiation.cc:407] Connection header received
0109 21:03:59.797088 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:59.797091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:59.797138 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:59.797231 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:59.797821 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:59.798405 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:59.799003 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:59.799132 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:59.801344 (+ 2212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.65:38697: BlockingRecv error: recv got EOF from 127.12.126.65:38697 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
Jan 09 21:03:59 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:03:59 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:03:59.999547 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:03:59.994707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:37583)
0109 21:03:59.995036 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0109 21:03:59.995052 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:03:59.995179 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:03:59.995469 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:03:59.995472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:03:59.995677 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:03:59.995869 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:59.995875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:59.996585 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:03:59.996588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:03:59.996958 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:03:59.996965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:03:59.997068 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:59.997634 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:03:59.997646 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:03:59.999389 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":130,"threads_started":1}
W20250109 21:03:59.999662 22531 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:03:59.994874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:37583 (local address 127.12.126.126:34575)
0109 21:03:59.995037 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0109 21:03:59.995043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:03:59.995132 (+ 89us) server_negotiation.cc:407] Connection header received
0109 21:03:59.995308 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:03:59.995313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:03:59.995364 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:03:59.995441 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:03:59.995987 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:59.996479 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:03:59.997065 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:03:59.997263 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:03:59.999541 (+ 2278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.66:37583: BlockingRecv error: recv got EOF from 127.12.126.66:37583 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jan 09 21:04:00 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:00 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:00.185642 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:00.180685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.67:34463)
0109 21:04:00.180976 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:00.180992 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:00.181122 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:00.181378 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:00.181382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:00.181584 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:00.181777 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:00.181783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:00.182493 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:00.182496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:00.182924 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:00.182932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:00.183040 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:00.183692 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:00.183712 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:00.185529 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":104,"threads_started":1}
W20250109 21:04:00.185788 22531 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:00.180823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:34463 (local address 127.12.126.126:34575)
0109 21:04:00.180976 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:00.180981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:00.181069 (+ 88us) server_negotiation.cc:407] Connection header received
0109 21:04:00.181216 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:00.181220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:00.181269 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:00.181341 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:00.181900 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:00.182384 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:00.183037 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:00.183240 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:00.185654 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.67:34463: BlockingRecv error: recv got EOF from 127.12.126.67:34463 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jan 09 21:04:00 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:00 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:00.806867 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:00.802102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:34759)
0109 21:04:00.802405 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:00.802419 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:00.802521 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:00.802763 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:00.802765 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:00.803028 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:00.803232 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:00.803239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:00.803961 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:00.803963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:00.804320 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:00.804326 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:00.804410 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:00.804962 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:00.804975 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:00.806698 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":90,"threads_started":1}
W20250109 21:04:00.807003 22535 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:00.802174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:34759 (local address 127.12.126.126:34575)
0109 21:04:00.802432 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:00.802436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:00.802484 (+ 48us) server_negotiation.cc:407] Connection header received
0109 21:04:00.802602 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:00.802604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:00.802651 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:00.802733 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:00.803365 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:00.803870 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:00.804405 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:00.804535 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:00.806887 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.65:34759: BlockingRecv error: recv got EOF from 127.12.126.65:34759 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
Jan 09 21:04:01 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:01 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:01.005251 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:01.000275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:36107)
0109 21:04:01.000570 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:01.000583 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:01.000694 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:01.000926 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:01.000928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:01.001146 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:01.001368 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:01.001374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:01.002064 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:01.002067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:01.002491 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:01.002506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:01.002612 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:01.003241 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:01.003258 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:01.005060 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
W20250109 21:04:01.005316 22535 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:01.000357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:36107 (local address 127.12.126.126:34575)
0109 21:04:01.000513 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:01.000519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:01.000642 (+ 123us) server_negotiation.cc:407] Connection header received
0109 21:04:01.000776 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:01.000779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:01.000830 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:01.000902 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:01.001476 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:01.001961 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:01.002601 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:01.002784 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:01.005194 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.66:36107: BlockingRecv error: recv got EOF from 127.12.126.66:36107 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jan 09 21:04:01 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:01 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:01.191255 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:01.186377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.67:51193)
0109 21:04:01.186710 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:01.186722 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:01.186864 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:01.187149 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:01.187152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:01.187361 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:01.187561 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:01.187568 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:01.188280 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:01.188293 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:01.188669 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:01.188676 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:01.188790 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:01.189389 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:01.189402 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:01.191131 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":174,"threads_started":1}
W20250109 21:04:01.191373 22535 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:01.186390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:51193 (local address 127.12.126.126:34575)
0109 21:04:01.186529 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:01.186540 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:01.186816 (+ 276us) server_negotiation.cc:407] Connection header received
0109 21:04:01.186956 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:01.186959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:01.187044 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:01.187114 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:01.187682 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:01.188173 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:01.188785 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:01.188987 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:01.191270 (+ 2283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.67:51193: BlockingRecv error: recv got EOF from 127.12.126.67:51193 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jan 09 21:04:01 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:01 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:01.812414 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:01.807761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:53345)
0109 21:04:01.808061 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:01.808078 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:01.808184 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:01.808433 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:01.808435 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:01.808629 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:01.808828 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:01.808833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:01.809531 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:01.809534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:01.809919 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:01.809926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:01.810014 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:01.810587 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:01.810599 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:01.812295 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":87,"threads_started":1}
W20250109 21:04:01.812599 22539 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:01.807807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:53345 (local address 127.12.126.126:34575)
0109 21:04:01.808056 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:01.808060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:01.808139 (+ 79us) server_negotiation.cc:407] Connection header received
0109 21:04:01.808277 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:01.808279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:01.808334 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:01.808408 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:01.808919 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:01.809433 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:01.810005 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:01.810124 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:01.812448 (+ 2324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.65:53345: BlockingRecv error: recv got EOF from 127.12.126.65:53345 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":66,"threads_started":1}
Jan 09 21:04:02 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:02 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:02.011492 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:02.006183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:37619)
0109 21:04:02.006464 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:02.006478 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:02.006565 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:02.007056 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:02.007059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:02.007438 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:02.007687 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:02.007695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:02.008422 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:02.008426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:02.008830 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:02.008838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:02.008952 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:02.009528 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:02.009542 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:02.011329 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
W20250109 21:04:02.011749 22539 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:02.006674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:37619 (local address 127.12.126.126:34575)
0109 21:04:02.006854 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:02.006858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:02.006869 (+ 11us) server_negotiation.cc:407] Connection header received
0109 21:04:02.006901 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:02.006903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:02.006945 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:02.007018 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:02.007818 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:02.008308 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:02.008947 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:02.009151 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:02.011520 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.66:37619: BlockingRecv error: recv got EOF from 127.12.126.66:37619 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jan 09 21:04:02 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:02 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:02.197247 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:02.192244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.67:49901)
0109 21:04:02.192543 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:02.192559 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:02.192683 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:02.192942 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:02.192945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:02.193148 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:02.193356 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:02.193362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:02.194123 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:02.194127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:02.194603 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:02.194615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:02.194736 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:02.195407 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:02.195423 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:02.197127 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":90,"threads_started":1}
W20250109 21:04:02.197384 22539 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:02.192390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:49901 (local address 127.12.126.126:34575)
0109 21:04:02.192554 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:02.192560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:02.192640 (+ 80us) server_negotiation.cc:407] Connection header received
0109 21:04:02.192780 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:02.192783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:02.192836 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:02.192908 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:02.193517 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:02.194010 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:02.194725 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:02.194952 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:02.197262 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.67:49901: BlockingRecv error: recv got EOF from 127.12.126.67:49901 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jan 09 21:04:02 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:02 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:02.823912 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:02.813445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:55003)
0109 21:04:02.813733 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:02.813747 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:02.813850 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:02.814096 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:02.814098 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:02.814276 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:02.814487 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:02.814493 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:02.815307 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:02.815311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:02.815868 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:02.815881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:02.816005 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:02.821778 (+ 5773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:02.821798 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:02.823776 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":96,"threads_started":1}
W20250109 21:04:02.824136 22543 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:02.813575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:55003 (local address 127.12.126.126:34575)
0109 21:04:02.813816 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:02.813820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:02.813835 (+ 15us) server_negotiation.cc:407] Connection header received
0109 21:04:02.813939 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:02.813942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:02.813990 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:02.814070 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:02.814590 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:02.815182 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:02.816009 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:02.816328 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:02.823990 (+ 7662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.65:55003: BlockingRecv error: recv got EOF from 127.12.126.65:55003 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":61,"threads_started":1}
Jan 09 21:04:03 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:03 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:03.017107 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:03.012340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:48245)
0109 21:04:03.012623 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:03.012635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:03.012738 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:03.012966 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:03.012968 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:03.013145 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:03.013347 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:03.013353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:03.014062 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:03.014064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:03.014451 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:03.014457 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:03.014553 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:03.015154 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:03.015167 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:03.016964 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":95,"threads_started":1}
W20250109 21:04:03.017259 22543 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:03.012404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:48245 (local address 127.12.126.126:34575)
0109 21:04:03.012528 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:03.012532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:03.012694 (+ 162us) server_negotiation.cc:407] Connection header received
0109 21:04:03.012811 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:03.012814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:03.012863 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:03.012944 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:03.013449 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:03.013972 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:03.014558 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:03.014698 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:03.017118 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.66:48245: BlockingRecv error: recv got EOF from 127.12.126.66:48245 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jan 09 21:04:03 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:03 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:03.205566 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:03.198169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.67:36803)
0109 21:04:03.198461 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:03.198474 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:03.198587 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:03.198874 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:03.198877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:03.199090 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:03.199312 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:03.199318 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:03.200264 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:03.200268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:03.200877 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:03.200888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:03.201023 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:03.203242 (+ 2219us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:03.203265 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:03.205404 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":88,"threads_started":1}
W20250109 21:04:03.205700 22543 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:03.198261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:36803 (local address 127.12.126.126:34575)
0109 21:04:03.198417 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:03.198422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:03.198542 (+ 120us) server_negotiation.cc:407] Connection header received
0109 21:04:03.198675 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:03.198678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:03.198751 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:03.198855 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:03.199415 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:03.200156 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:03.201008 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:03.201257 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:03.205580 (+ 4323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.67:36803: BlockingRecv error: recv got EOF from 127.12.126.67:36803 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jan 09 21:04:03 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:03 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:03.829687 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:03.824742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:52749)
0109 21:04:03.825036 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:03.825050 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:03.825152 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:03.825395 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:03.825398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:03.825604 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:03.825800 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:03.825806 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:03.826743 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:03.826746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:03.827216 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:03.827228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:03.827324 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:03.827902 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:03.827915 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:03.829568 (+ 1653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":71,"threads_started":1}
W20250109 21:04:03.829838 22547 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:03.824925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:52749 (local address 127.12.126.126:34575)
0109 21:04:03.825171 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:03.825175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:03.825190 (+ 15us) server_negotiation.cc:407] Connection header received
0109 21:04:03.825232 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:03.825234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:03.825279 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:03.825366 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:03.825913 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:03.826623 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:03.827320 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:03.827456 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:03.829711 (+ 2255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.65:52749: BlockingRecv error: recv got EOF from 127.12.126.65:52749 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
Jan 09 21:04:04 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:04 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:04.022514 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:04.017916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:45007)
0109 21:04:04.018184 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:04.018196 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:04.018298 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:04.018533 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:04.018535 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:04.018707 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:04.018980 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:04.018988 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:04.019678 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:04.019680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:04.020072 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:04.020078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:04.020163 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:04.020685 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:04.020698 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:04.022398 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":72,"threads_started":1}
W20250109 21:04:04.022667 22547 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:04.017989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:45007 (local address 127.12.126.126:34575)
0109 21:04:04.018125 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:04.018129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:04.018253 (+ 124us) server_negotiation.cc:407] Connection header received
0109 21:04:04.018369 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:04.018372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:04.018420 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:04.018494 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:04.019082 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:04.019592 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:04.020146 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:04.020260 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:04.022535 (+ 2275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.66:45007: BlockingRecv error: recv got EOF from 127.12.126.66:45007 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jan 09 21:04:04 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:04 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:04.211148 22549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:04.206464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.67:45079)
0109 21:04:04.206775 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:04.206788 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:04.206911 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:04.207155 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:04.207157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:04.207351 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:04.207595 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:04.207603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:04.208274 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:04.208276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:04.208669 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:04.208675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:04.208765 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:04.209331 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:04.209344 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:04.211028 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":99,"threads_started":1}
W20250109 21:04:04.211305 22547 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:04.206551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:45079 (local address 127.12.126.126:34575)
0109 21:04:04.206711 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:04.206717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:04.206872 (+ 155us) server_negotiation.cc:407] Connection header received
0109 21:04:04.207003 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:04.207006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:04.207059 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:04.207131 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:04.207696 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:04.208187 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:04.208748 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:04.208869 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:04.211163 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.67:45079: BlockingRecv error: recv got EOF from 127.12.126.67:45079 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jan 09 21:04:04 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:04 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:04.835424 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:04.830560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:35637)
0109 21:04:04.830879 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:04.830893 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:04.830989 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:04.831230 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:04.831232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:04.831408 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:04.831614 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:04.831621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:04.832339 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:04.832342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:04.832781 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:04.832791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:04.832899 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:04.833535 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:04.833556 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:04.835307 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
W20250109 21:04:04.835572 22551 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:04.830716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:35637 (local address 127.12.126.126:34575)
0109 21:04:04.830947 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:04.830951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:04.830965 (+ 14us) server_negotiation.cc:407] Connection header received
0109 21:04:04.831069 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:04.831072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:04.831119 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:04.831206 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:04.831709 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:04.832200 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:04.832891 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:04.833101 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:04.835443 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.65:35637: BlockingRecv error: recv got EOF from 127.12.126.65:35637 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
Jan 09 21:04:05 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:05 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:05.028223 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:05.023231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:44309)
0109 21:04:05.023519 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:05.023536 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:05.023652 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:05.023907 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:05.023910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:05.024109 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:05.024317 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:05.024324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:05.025204 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:05.025207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:05.025583 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:05.025590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:05.025696 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:05.026266 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:05.026280 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:05.028056 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
W20250109 21:04:05.028295 22551 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:05.023332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:44309 (local address 127.12.126.126:34575)
0109 21:04:05.023488 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:05.023493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:05.023600 (+ 107us) server_negotiation.cc:407] Connection header received
0109 21:04:05.023744 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:05.023747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:05.023809 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:05.023880 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:05.024435 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:05.025098 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:05.025685 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:05.025876 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:05.028188 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.66:44309: BlockingRecv error: recv got EOF from 127.12.126.66:44309 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jan 09 21:04:05 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:05 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:05.216562 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:05.211907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.67:49219)
0109 21:04:05.212235 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:05.212249 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:05.212356 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:05.212588 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:05.212590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:05.212762 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:05.212957 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:05.212962 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:05.213669 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:05.213671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:05.214039 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:05.214045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:05.214130 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:05.214721 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:05.214734 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:05.216433 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":85,"threads_started":1}
W20250109 21:04:05.216768 22551 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:05.212098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:49219 (local address 127.12.126.126:34575)
0109 21:04:05.212237 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:05.212240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:05.212313 (+ 73us) server_negotiation.cc:407] Connection header received
0109 21:04:05.212431 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:05.212434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:05.212483 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:05.212556 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:05.213041 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:05.213580 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:05.214116 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:05.214241 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:05.216602 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.67:49219: BlockingRecv error: recv got EOF from 127.12.126.67:49219 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jan 09 21:04:05 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:05 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:05.840903 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:05.836195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:53637)
0109 21:04:05.836495 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:05.836514 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:05.836609 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:05.836829 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:05.836831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:05.837010 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:05.837213 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:05.837219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:05.837913 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:05.837916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:05.838284 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:05.838290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:05.838374 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:05.839016 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:05.839029 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:05.840765 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
W20250109 21:04:05.841063 22555 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:05.836308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:53637 (local address 127.12.126.126:34575)
0109 21:04:05.836536 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:05.836540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:05.836557 (+ 17us) server_negotiation.cc:407] Connection header received
0109 21:04:05.836687 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:05.836689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:05.836734 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:05.836811 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:05.837309 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:05.837828 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:05.838450 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:05.838639 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:05.840926 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.65:53637: BlockingRecv error: recv got EOF from 127.12.126.65:53637 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":58,"threads_started":1}
Jan 09 21:04:06 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:06 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:06.034420 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:06.029170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:54433)
0109 21:04:06.029512 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:06.029530 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:06.029679 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:06.029954 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:06.029957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:06.030284 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:06.030502 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:06.030509 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:06.031323 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:06.031327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:06.031708 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:06.031716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:06.031827 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:06.032414 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:06.032429 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:06.034254 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":101,"threads_started":1}
W20250109 21:04:06.034526 22555 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:06.029295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:54433 (local address 127.12.126.126:34575)
0109 21:04:06.029517 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:06.029524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:06.029611 (+ 87us) server_negotiation.cc:407] Connection header received
0109 21:04:06.029783 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:06.029787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:06.029842 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:06.029921 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:06.030655 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:06.031215 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:06.031819 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:06.032018 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:06.034400 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.66:54433: BlockingRecv error: recv got EOF from 127.12.126.66:54433 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Jan 09 21:04:06 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:06 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:06.222218 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:06.217416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.67:47305)
0109 21:04:06.217726 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:06.217740 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:06.217855 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:06.218099 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:06.218101 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:06.218282 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:06.218485 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:06.218490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:06.219248 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:06.219252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:06.219634 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:06.219642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:06.219754 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:06.220304 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:06.220318 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:06.222034 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":94,"threads_started":1}
W20250109 21:04:06.222324 22555 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:06.217514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:47305 (local address 127.12.126.126:34575)
0109 21:04:06.217673 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:06.217679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:06.217816 (+ 137us) server_negotiation.cc:407] Connection header received
0109 21:04:06.217952 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:06.217954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:06.218006 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:06.218074 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:06.218603 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:06.219127 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:06.219742 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:06.219931 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:06.222173 (+ 2242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.67:47305: BlockingRecv error: recv got EOF from 127.12.126.67:47305 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jan 09 21:04:06 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:06 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:06.846709 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:06.841860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:53289)
0109 21:04:06.842157 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:06.842175 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:06.842302 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:06.842599 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:06.842602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:06.842845 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:06.843086 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:06.843094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:06.843824 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:06.843827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:06.844199 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:06.844206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:06.844309 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:06.844873 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:06.844886 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:06.846587 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
W20250109 21:04:06.846884 22559 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:06.841962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:53289 (local address 127.12.126.126:34575)
0109 21:04:06.842252 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:06.842257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:06.842275 (+ 18us) server_negotiation.cc:407] Connection header received
0109 21:04:06.842415 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:06.842419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:06.842477 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:06.842578 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:06.843200 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:06.843721 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:06.844308 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:06.844512 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:06.846712 (+ 2200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.65:53289: BlockingRecv error: recv got EOF from 127.12.126.65:53289 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":92,"threads_started":1}
Jan 09 21:04:07 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:07 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:07.040134 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:07.035390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:44611)
0109 21:04:07.035685 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:07.035698 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:07.035793 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:07.036013 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:07.036015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:07.036199 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:07.036408 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:07.036414 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:07.037086 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:07.037089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:07.037478 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:07.037485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:07.037587 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:07.038154 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:07.038167 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:07.039973 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":94,"threads_started":1}
W20250109 21:04:07.040246 22559 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:07.035498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:44611 (local address 127.12.126.126:34575)
0109 21:04:07.035655 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:07.035660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:07.035761 (+ 101us) server_negotiation.cc:407] Connection header received
0109 21:04:07.035871 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:07.035873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:07.035920 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:07.035993 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:07.036501 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:07.036993 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:07.037570 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:07.037688 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:07.040131 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.66:44611: BlockingRecv error: recv got EOF from 127.12.126.66:44611 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jan 09 21:04:07 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:07 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:07.229745 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:07.223167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.67:55053)
0109 21:04:07.223419 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:07.223434 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:07.223524 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:07.223797 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:07.223800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:07.224021 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:07.224299 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:07.224308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:07.225160 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:07.225164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:07.225733 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:07.225747 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:07.225859 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:07.227275 (+ 1416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:07.227294 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:07.229565 (+ 2271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":67,"threads_started":1}
W20250109 21:04:07.229843 22559 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:07.223289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:55053 (local address 127.12.126.126:34575)
0109 21:04:07.223461 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:07.223466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:07.223480 (+ 14us) server_negotiation.cc:407] Connection header received
0109 21:04:07.223617 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:07.223620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:07.223680 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:07.223778 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:07.224403 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:07.225046 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:07.225853 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:07.226066 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:07.229715 (+ 3649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.67:55053: BlockingRecv error: recv got EOF from 127.12.126.67:55053 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jan 09 21:04:07 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:07 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:07.859457 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:07.847675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:52553)
0109 21:04:07.847934 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:07.847949 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:07.848041 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:07.848285 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:07.848287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:07.848469 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:07.848667 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:07.848673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:07.849397 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:07.849400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:07.849834 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:07.849844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:07.849948 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:07.857411 (+ 7463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:07.857430 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:07.859326 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
W20250109 21:04:07.859761 22563 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:07.847764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:52553 (local address 127.12.126.126:34575)
0109 21:04:07.847997 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:07.848001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:07.848016 (+ 15us) server_negotiation.cc:407] Connection header received
0109 21:04:07.848120 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:07.848122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:07.848172 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:07.848265 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:07.848765 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:07.849303 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:07.849937 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:07.850099 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:07.859614 (+ 9515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.65:52553: BlockingRecv error: recv got EOF from 127.12.126.65:52553 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
Jan 09 21:04:08 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:08 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:08.045331 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:08.040853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:34503)
0109 21:04:08.041085 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:08.041097 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:08.041191 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:08.041424 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:08.041426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:08.041609 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:08.041807 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:08.041812 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:08.042469 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:08.042472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:08.042884 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:08.042892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:08.042980 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:08.043509 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:08.043522 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:08.045208 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
W20250109 21:04:08.045502 22563 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:08.040912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:34503 (local address 127.12.126.126:34575)
0109 21:04:08.041028 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:08.041031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:08.041151 (+ 120us) server_negotiation.cc:407] Connection header received
0109 21:04:08.041264 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:08.041266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:08.041334 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:08.041404 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:08.041896 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:08.042390 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:08.042970 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:08.043102 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:08.045353 (+ 2251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.66:34503: BlockingRecv error: recv got EOF from 127.12.126.66:34503 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jan 09 21:04:08 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:08 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:08.235814 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:08.230861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.67:49719)
0109 21:04:08.231108 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:08.231120 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:08.231223 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:08.231517 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:08.231519 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:08.231753 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:08.231982 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:08.231989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:08.232696 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:08.232699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:08.233072 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:08.233078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:08.233169 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:08.233725 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:08.233737 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:08.235649 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":79,"threads_started":1}
W20250109 21:04:08.235935 22563 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:08.230921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:49719 (local address 127.12.126.126:34575)
0109 21:04:08.231046 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:08.231049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:08.231181 (+ 132us) server_negotiation.cc:407] Connection header received
0109 21:04:08.231331 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:08.231334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:08.231385 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:08.231459 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:08.232097 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:08.232600 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:08.233148 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:08.233270 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:08.235823 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.67:49719: BlockingRecv error: recv got EOF from 127.12.126.67:49719 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jan 09 21:04:08 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:08 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:08.865106 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:08.860269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:40019)
0109 21:04:08.860584 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:08.860599 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:08.860692 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:08.860920 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:08.860922 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:08.861112 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:08.861337 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:08.861343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:08.862106 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:08.862109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:08.862517 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:08.862525 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:08.862624 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:08.863223 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:08.863237 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:08.864948 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":77,"threads_started":1}
W20250109 21:04:08.865227 22567 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:08.860369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:40019 (local address 127.12.126.126:34575)
0109 21:04:08.860640 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:08.860644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:08.860660 (+ 16us) server_negotiation.cc:407] Connection header received
0109 21:04:08.860769 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:08.860772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:08.860818 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:08.860900 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:08.861436 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:08.861995 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:08.862603 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:08.862729 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:08.865100 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.65:40019: BlockingRecv error: recv got EOF from 127.12.126.65:40019 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
Jan 09 21:04:09 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:09 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:09.051154 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:09.046139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:52293)
0109 21:04:09.046509 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:09.046526 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:09.046647 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:09.046956 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:09.046959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:09.047174 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:09.047375 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:09.047382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:09.048090 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:09.048093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:09.048488 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:09.048496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:09.048600 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:09.049183 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:09.049194 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:09.050966 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":113,"threads_started":1}
W20250109 21:04:09.051241 22567 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:09.046217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:52293 (local address 127.12.126.126:34575)
0109 21:04:09.046383 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:09.046388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:09.046614 (+ 226us) server_negotiation.cc:407] Connection header received
0109 21:04:09.046765 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:09.046768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:09.046853 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:09.046931 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:09.047505 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:09.047984 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:09.048597 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:09.048796 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:09.051127 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.66:52293: BlockingRecv error: recv got EOF from 127.12.126.66:52293 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jan 09 21:04:09 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:09 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:09.241796 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:09.236815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.67:54149)
0109 21:04:09.237108 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:09.237124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:09.237245 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:09.237498 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:09.237502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:09.237705 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:09.237903 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:09.237908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:09.238617 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:09.238621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:09.239022 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:09.239029 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:09.239139 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:09.239822 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:09.239835 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:09.241630 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":92,"threads_started":1}
W20250109 21:04:09.241875 22567 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:09.236941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:54149 (local address 127.12.126.126:34575)
0109 21:04:09.237089 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:09.237094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:09.237192 (+ 98us) server_negotiation.cc:407] Connection header received
0109 21:04:09.237345 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:09.237348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:09.237395 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:09.237464 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:09.238016 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:09.238514 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:09.239127 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:09.239312 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:09.241762 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.67:54149: BlockingRecv error: recv got EOF from 127.12.126.67:54149 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jan 09 21:04:09 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:09 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:09.870606 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:09.865920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:39639)
0109 21:04:09.866174 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:09.866187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:09.866276 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:09.866499 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:09.866501 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:09.866693 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:09.866921 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:09.866927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:09.867611 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:09.867613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:09.867975 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:09.867980 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:09.868073 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:09.868626 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:09.868640 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:09.870445 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
W20250109 21:04:09.870723 22571 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:09.866084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:39639 (local address 127.12.126.126:34575)
0109 21:04:09.866299 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:09.866303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:09.866316 (+ 13us) server_negotiation.cc:407] Connection header received
0109 21:04:09.866352 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:09.866354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:09.866397 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:09.866479 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:09.867011 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:09.867527 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:09.868055 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:09.868187 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:09.870589 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.65:39639: BlockingRecv error: recv got EOF from 127.12.126.65:39639 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":67,"threads_started":1}
Jan 09 21:04:10 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:10 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:10.056960 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:10.052140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:51331)
0109 21:04:10.052405 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:10.052419 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:10.052504 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:10.052783 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:10.052785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:10.052963 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:10.053164 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:10.053169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:10.053994 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:10.053996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:10.054367 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:10.054372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:10.054459 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:10.055075 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:10.055089 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:10.056798 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":74,"threads_started":1}
W20250109 21:04:10.057066 22571 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:10.052313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:51331 (local address 127.12.126.126:34575)
0109 21:04:10.052439 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:10.052443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:10.052489 (+ 46us) server_negotiation.cc:407] Connection header received
0109 21:04:10.052628 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:10.052630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:10.052679 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:10.052761 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:10.053351 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:10.053907 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:10.054449 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:10.054571 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:10.056942 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.66:51331: BlockingRecv error: recv got EOF from 127.12.126.66:51331 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jan 09 21:04:10 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:10 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:10.247485 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:10.242731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.67:37877)
0109 21:04:10.243039 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:10.243052 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:10.243156 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:10.243400 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:10.243402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:10.243589 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:10.243788 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:10.243794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:10.244519 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:10.244523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:10.244934 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:10.244943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:10.245035 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:10.245624 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:10.245643 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:10.247367 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":131,"threads_started":1}
W20250109 21:04:10.247655 22571 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:10.242865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:37877 (local address 127.12.126.126:34575)
0109 21:04:10.242991 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:10.242995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:10.243118 (+ 123us) server_negotiation.cc:407] Connection header received
0109 21:04:10.243235 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:10.243238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:10.243292 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:10.243377 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:10.243889 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:10.244401 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:10.245035 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:10.245256 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:10.247506 (+ 2250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.67:37877: BlockingRecv error: recv got EOF from 127.12.126.67:37877 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jan 09 21:04:10 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:10 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:10.876171 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:10.871460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:57559)
0109 21:04:10.871746 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:10.871760 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:10.871852 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:10.872085 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:10.872087 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:10.872265 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:10.872478 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:10.872484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:10.873179 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:10.873181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:10.873577 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:10.873583 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:10.873681 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:10.874212 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:10.874226 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:10.876041 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":95,"threads_started":1}
W20250109 21:04:10.876397 22575 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:10.871565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:57559 (local address 127.12.126.126:34575)
0109 21:04:10.871834 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:10.871837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:10.871850 (+ 13us) server_negotiation.cc:407] Connection header received
0109 21:04:10.871930 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:10.871932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:10.871978 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:10.872060 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:10.872580 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:10.873092 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:10.873683 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:10.873819 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:10.876247 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.65:57559: BlockingRecv error: recv got EOF from 127.12.126.65:57559 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":68,"threads_started":1}
Jan 09 21:04:11 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:11 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:11.062467 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:11.057735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:51877)
0109 21:04:11.058075 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:11.058089 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:11.058190 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:11.058417 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:11.058419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:11.058614 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:11.058868 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:11.058875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:11.059571 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:11.059574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:11.059956 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:11.059963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:11.060053 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:11.060572 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:11.060584 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:11.062347 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":77,"threads_started":1}
W20250109 21:04:11.062602 22575 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:11.057782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:51877 (local address 127.12.126.126:34575)
0109 21:04:11.057923 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:11.057927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:11.058149 (+ 222us) server_negotiation.cc:407] Connection header received
0109 21:04:11.058272 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:11.058274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:11.058332 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:11.058395 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:11.058964 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:11.059474 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:11.060037 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:11.060161 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:11.062498 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.66:51877: BlockingRecv error: recv got EOF from 127.12.126.66:51877 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jan 09 21:04:11 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:11 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.67@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:11.252779 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:11.248213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.67:45857)
0109 21:04:11.248490 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:11.248504 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:11.248607 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:11.248835 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:11.248837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:11.249018 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:11.249222 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:11.249227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:11.249905 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:11.249907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:11.250290 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:11.250296 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:11.250386 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:11.250995 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:11.251008 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:11.252655 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":98,"threads_started":1}
W20250109 21:04:11.252936 22575 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:11.248285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:45857 (local address 127.12.126.126:34575)
0109 21:04:11.248412 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:11.248415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:11.248569 (+ 154us) server_negotiation.cc:407] Connection header received
0109 21:04:11.248683 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:11.248685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:11.248737 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:11.248812 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:11.249337 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:11.249825 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:11.250370 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:11.250489 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:11.252812 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.67:45857: BlockingRecv error: recv got EOF from 127.12.126.67:45857 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jan 09 21:04:11 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:11 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.65@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:11.881816 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:11.877090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.65:45449)
0109 21:04:11.877354 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:11.877368 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:11.877454 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:11.877736 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:11.877738 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:11.877934 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:11.878133 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:11.878138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:11.878909 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:11.878911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:11.879289 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:11.879295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:11.879380 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:11.879944 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:11.879958 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:11.881697 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":84,"threads_started":1}
W20250109 21:04:11.881985 22579 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:11.877175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:45449 (local address 127.12.126.126:34575)
0109 21:04:11.877428 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:11.877431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:11.877445 (+ 14us) server_negotiation.cc:407] Connection header received
0109 21:04:11.877573 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:11.877576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:11.877631 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:11.877716 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:11.878254 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:11.878805 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:11.879363 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:11.879493 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:11.881848 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.65:45449: BlockingRecv error: recv got EOF from 127.12.126.65:45449 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":63,"threads_started":1}
Jan 09 21:04:12 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
Jan 09 21:04:12 dist-test-slave-g0zb krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.126.66@KRBTEST.COM for oryx/127.12.126.126@KRBTEST.COM, Server not found in Kerberos database
I20250109 21:04:12.067930 22580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:12.063362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:34575 (local address 127.12.126.66:38021)
0109 21:04:12.063640 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:12.063653 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:12.063753 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:12.063965 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:12.063967 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:12.064142 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:12.064340 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:12.064345 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:12.065002 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:12.065004 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:12.065365 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:12.065372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:12.065456 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:12.066056 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:12.066070 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:12.067813 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.126.126:34575: Server oryx/127.12.126.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":99,"threads_started":1}
W20250109 21:04:12.068087 22579 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:12.063458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:38021 (local address 127.12.126.126:34575)
0109 21:04:12.063614 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:12.063618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:12.063714 (+ 96us) server_negotiation.cc:407] Connection header received
0109 21:04:12.063822 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:12.063824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:12.063873 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:12.063942 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:12.064427 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:12.064918 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:12.065434 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:12.065550 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:12.067950 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.126.66:38021: BlockingRecv error: recv got EOF from 127.12.126.66:38021 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20250109 21:04:12.153381 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 22104
I20250109 21:04:12.159571 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 22256
I20250109 21:04:12.165310 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 22388
I20250109 21:04:12.170924 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 22034
2025-01-09T21:04:12Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17890 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2025-01-09T21:04:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:04:12Z Disabled control of system clock
I20250109 21:04:12.201167 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:41405
--webserver_interface=127.12.126.126
--webserver_port=0
--builtin_ntp_servers=127.12.126.84:42311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:41405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20250109 21:04:12.308545 22584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:04:12.308795 22584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:04:12.308838 22584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:04:12.311995 22584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:04:12.312065 22584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:04:12.312095 22584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:04:12.312114 22584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:04:12.312136 22584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:04:12.315874 22584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:41405
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:41405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:04:12.316985 22584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:04:12.318171 22584 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:04:12.325662 22589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:12.326397 22592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:04:12.327234 22584 server_base.cc:1034] running on GCE node
W20250109 21:04:12.327888 22590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:04:12.328413 22584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:04:12.329331 22584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:04:12.330475 22584 hybrid_clock.cc:648] HybridClock initialized: now 1736456652330461 us; error 31 us; skew 500 ppm
I20250109 21:04:12.332511 22584 webserver.cc:458] Webserver started at http://127.12.126.126:42759/ using document root <none> and password file <none>
I20250109 21:04:12.333098 22584 fs_manager.cc:362] Metadata directory not provided
I20250109 21:04:12.333158 22584 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:04:12.333338 22584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:04:12.335220 22584 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "dc08390ec19640f69eac6cc9d8671ba5"
format_stamp: "Formatted at 2025-01-09 21:04:12 on dist-test-slave-g0zb"
server_key: "bbc01bd28b259dbaabe4bed4a3aec4e4"
server_key_iv: "b26108b91eb5496ce71ca908c3c3a59b"
server_key_version: "encryptionkey@0"
I20250109 21:04:12.335718 22584 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "dc08390ec19640f69eac6cc9d8671ba5"
format_stamp: "Formatted at 2025-01-09 21:04:12 on dist-test-slave-g0zb"
server_key: "bbc01bd28b259dbaabe4bed4a3aec4e4"
server_key_iv: "b26108b91eb5496ce71ca908c3c3a59b"
server_key_version: "encryptionkey@0"
I20250109 21:04:12.339051 22584 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:04:12.341181 22598 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:12.342188 22584 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:04:12.342311 22584 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "dc08390ec19640f69eac6cc9d8671ba5"
format_stamp: "Formatted at 2025-01-09 21:04:12 on dist-test-slave-g0zb"
server_key: "bbc01bd28b259dbaabe4bed4a3aec4e4"
server_key_iv: "b26108b91eb5496ce71ca908c3c3a59b"
server_key_version: "encryptionkey@0"
I20250109 21:04:12.342401 22584 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:04:12.356370 22584 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:04:12.357012 22584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:04:12.357172 22584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:04:12.363831 22584 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:41405
I20250109 21:04:12.363840 22650 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:41405 every 8 connection(s)
I20250109 21:04:12.364867 22584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:04:12.366016 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 22584
I20250109 21:04:12.366124 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:04:12.366362 12793 external_mini_cluster.cc:1453] Setting key 91ea31f8a10fb79081ce94fe8984eece
I20250109 21:04:12.368006 22651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:04:12.373411 22651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5: Bootstrap starting.
I20250109 21:04:12.373979 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:12.367511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35532 (local address 127.12.126.126:41405)
0109 21:04:12.368177 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:12.368191 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:12.368236 (+ 45us) server_negotiation.cc:407] Connection header received
0109 21:04:12.368824 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:12.368840 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:12.369201 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:12.369533 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:12.370029 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:12.370790 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:12.371464 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:12.371770 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:12.372309 (+ 539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:04:12.372369 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:04:12.372381 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:04:12.372758 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:04:12.372798 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:04:12.372814 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:04:12.372911 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:12.373046 (+ 135us) server_negotiation.cc:299] Negotiation successful
0109 21:04:12.373277 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":128,"threads_started":1}
I20250109 21:04:12.375854 22651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5: Neither blocks nor log segments found. Creating new log.
I20250109 21:04:12.376658 22651 log.cc:826] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5: Log is configured to *not* fsync() on all Append() calls
I20250109 21:04:12.378449 22651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5: No bootstrap required, opened a new log
I20250109 21:04:12.381727 22651 raft_consensus.cc:357] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc08390ec19640f69eac6cc9d8671ba5" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41405 } }
I20250109 21:04:12.381928 22651 raft_consensus.cc:383] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:04:12.381981 22651 raft_consensus.cc:738] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc08390ec19640f69eac6cc9d8671ba5, State: Initialized, Role: FOLLOWER
I20250109 21:04:12.382428 22651 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc08390ec19640f69eac6cc9d8671ba5" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41405 } }
I20250109 21:04:12.382558 22651 raft_consensus.cc:397] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:04:12.382624 22651 raft_consensus.cc:491] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:04:12.382735 22651 raft_consensus.cc:3054] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:04:12.383659 22651 raft_consensus.cc:513] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc08390ec19640f69eac6cc9d8671ba5" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41405 } }
I20250109 21:04:12.384009 22651 leader_election.cc:304] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc08390ec19640f69eac6cc9d8671ba5; no voters:
I20250109 21:04:12.384325 22651 leader_election.cc:290] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:04:12.384429 22656 raft_consensus.cc:2798] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:04:12.384641 22656 raft_consensus.cc:695] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5 [term 1 LEADER]: Becoming Leader. State: Replica: dc08390ec19640f69eac6cc9d8671ba5, State: Running, Role: LEADER
I20250109 21:04:12.384984 22656 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc08390ec19640f69eac6cc9d8671ba5" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41405 } }
I20250109 21:04:12.385373 22651 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:04:12.386433 22657 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc08390ec19640f69eac6cc9d8671ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc08390ec19640f69eac6cc9d8671ba5" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41405 } } }
I20250109 21:04:12.386564 22657 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5 [sys.catalog]: This master's current role is: LEADER
I20250109 21:04:12.386987 22665 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:04:12.386945 22658 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc08390ec19640f69eac6cc9d8671ba5. Latest consensus state: current_term: 1 leader_uuid: "dc08390ec19640f69eac6cc9d8671ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc08390ec19640f69eac6cc9d8671ba5" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41405 } } }
I20250109 21:04:12.387395 22658 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5 [sys.catalog]: This master's current role is: LEADER
I20250109 21:04:12.390143 22665 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:04:12.395509 22665 catalog_manager.cc:1348] Generated new cluster ID: 0a12586472cd472f94e1ea68af9ba6fc
I20250109 21:04:12.395576 22665 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:04:12.408878 22665 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:04:12.410181 22665 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:04:12.423966 22665 catalog_manager.cc:5899] T 00000000000000000000000000000000 P dc08390ec19640f69eac6cc9d8671ba5: Generated new TSK 0
I20250109 21:04:12.424880 22665 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:04:12.437402 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:41405
--builtin_ntp_servers=127.12.126.84:42311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20250109 21:04:12.548002 22675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:04:12.548240 22675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:04:12.548290 22675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:04:12.551527 22675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:04:12.551594 22675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:04:12.551666 22675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:04:12.555471 22675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:41405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:04:12.556674 22675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:04:12.557826 22675 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:04:12.566146 22680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:12.566941 22681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:12.567546 22683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:04:12.568532 22675 server_base.cc:1034] running on GCE node
I20250109 21:04:12.568972 22675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:04:12.569588 22675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:04:12.570729 22675 hybrid_clock.cc:648] HybridClock initialized: now 1736456652570706 us; error 40 us; skew 500 ppm
I20250109 21:04:12.572791 22675 webserver.cc:458] Webserver started at http://127.12.126.65:35591/ using document root <none> and password file <none>
I20250109 21:04:12.573372 22675 fs_manager.cc:362] Metadata directory not provided
I20250109 21:04:12.573431 22675 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:04:12.573625 22675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:04:12.575518 22675 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "f03fca474aa04ea9bcb1aa59db2ddab4"
format_stamp: "Formatted at 2025-01-09 21:04:12 on dist-test-slave-g0zb"
server_key: "7b3648402553f7c0473ba7cec20d94ee"
server_key_iv: "b64d1e447f78e3e315243e4bc176da1e"
server_key_version: "encryptionkey@0"
I20250109 21:04:12.576007 22675 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "f03fca474aa04ea9bcb1aa59db2ddab4"
format_stamp: "Formatted at 2025-01-09 21:04:12 on dist-test-slave-g0zb"
server_key: "7b3648402553f7c0473ba7cec20d94ee"
server_key_iv: "b64d1e447f78e3e315243e4bc176da1e"
server_key_version: "encryptionkey@0"
I20250109 21:04:12.579252 22675 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:04:12.581351 22689 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:12.582316 22675 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:04:12.582432 22675 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "f03fca474aa04ea9bcb1aa59db2ddab4"
format_stamp: "Formatted at 2025-01-09 21:04:12 on dist-test-slave-g0zb"
server_key: "7b3648402553f7c0473ba7cec20d94ee"
server_key_iv: "b64d1e447f78e3e315243e4bc176da1e"
server_key_version: "encryptionkey@0"
I20250109 21:04:12.582527 22675 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:04:12.593274 22675 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:04:12.593864 22675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:04:12.594013 22675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:04:12.594513 22675 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:04:12.595405 22675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:04:12.595451 22675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:12.595528 22675 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:04:12.595552 22675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:12.603713 22675 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:35087
I20250109 21:04:12.603744 22802 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:35087 every 8 connection(s)
I20250109 21:04:12.604629 22675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:04:12.612228 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:12.606253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:56193 (local address 127.12.126.126:41405)
0109 21:04:12.606391 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:12.606395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:12.607307 (+ 912us) server_negotiation.cc:407] Connection header received
0109 21:04:12.608139 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:12.608142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:12.608202 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:12.608279 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:12.609638 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:12.610152 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:12.610770 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:12.610976 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:12.611751 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:04:12.611771 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:04:12.611774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:04:12.611786 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:04:12.611804 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:04:12.611811 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:04:12.611874 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:12.612016 (+ 142us) server_negotiation.cc:299] Negotiation successful
0109 21:04:12.612081 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250109 21:04:12.612895 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:12.606532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:41405 (local address 127.12.126.65:56193)
0109 21:04:12.607178 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:12.607210 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:12.607915 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:12.608450 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:12.608458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:12.608531 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:12.609496 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:12.609510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:12.610273 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:12.610275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:12.610661 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:12.610668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:12.610877 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:12.611512 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:04:12.611514 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:12.611546 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:04:12.611626 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:04:12.611884 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:04:12.611893 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:04:12.611899 (+ 6us) client_negotiation.cc:769] Sending connection context
0109 21:04:12.612016 (+ 117us) client_negotiation.cc:240] Negotiation successful
0109 21:04:12.612227 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":124,"threads_started":1}
I20250109 21:04:12.613252 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 22675
I20250109 21:04:12.613373 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:04:12.613691 12793 external_mini_cluster.cc:1453] Setting key 511c626a0f79ddea6d118de4e827bec4
I20250109 21:04:12.614058 22803 heartbeater.cc:346] Connected to a master server at 127.12.126.126:41405
I20250109 21:04:12.614289 22803 heartbeater.cc:463] Registering TS with master...
I20250109 21:04:12.614873 22803 heartbeater.cc:510] Master 127.12.126.126:41405 requested a full tablet report, sending...
I20250109 21:04:12.615586 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:41405
--builtin_ntp_servers=127.12.126.84:42311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20250109 21:04:12.616338 22615 ts_manager.cc:194] Registered new tserver with Master: f03fca474aa04ea9bcb1aa59db2ddab4 (127.12.126.65:35087)
I20250109 21:04:12.617668 22615 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.126.65:56193
W20250109 21:04:12.723045 22806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:04:12.723289 22806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:04:12.723338 22806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:04:12.726464 22806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:04:12.726526 22806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:04:12.726595 22806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:04:12.730233 22806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:41405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:04:12.731408 22806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:04:12.732537 22806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:04:12.740765 22814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:12.741472 22812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:12.742342 22811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:04:12.743062 22806 server_base.cc:1034] running on GCE node
I20250109 21:04:12.743469 22806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:04:12.744050 22806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:04:12.745193 22806 hybrid_clock.cc:648] HybridClock initialized: now 1736456652745179 us; error 32 us; skew 500 ppm
I20250109 21:04:12.747190 22806 webserver.cc:458] Webserver started at http://127.12.126.66:37419/ using document root <none> and password file <none>
I20250109 21:04:12.747776 22806 fs_manager.cc:362] Metadata directory not provided
I20250109 21:04:12.747835 22806 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:04:12.748009 22806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:04:12.749861 22806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "c3e8fbf842f24253a2a07749f55ea250"
format_stamp: "Formatted at 2025-01-09 21:04:12 on dist-test-slave-g0zb"
server_key: "7b9d626fb7118e3d2187a723417b335b"
server_key_iv: "22dbad8a3478d7345c46475bad482e23"
server_key_version: "encryptionkey@0"
I20250109 21:04:12.750351 22806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "c3e8fbf842f24253a2a07749f55ea250"
format_stamp: "Formatted at 2025-01-09 21:04:12 on dist-test-slave-g0zb"
server_key: "7b9d626fb7118e3d2187a723417b335b"
server_key_iv: "22dbad8a3478d7345c46475bad482e23"
server_key_version: "encryptionkey@0"
I20250109 21:04:12.753751 22806 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250109 21:04:12.755939 22820 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:12.756912 22806 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:04:12.757017 22806 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "c3e8fbf842f24253a2a07749f55ea250"
format_stamp: "Formatted at 2025-01-09 21:04:12 on dist-test-slave-g0zb"
server_key: "7b9d626fb7118e3d2187a723417b335b"
server_key_iv: "22dbad8a3478d7345c46475bad482e23"
server_key_version: "encryptionkey@0"
I20250109 21:04:12.757105 22806 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:04:12.771749 22806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:04:12.772347 22806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:04:12.772505 22806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:04:12.773023 22806 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:04:12.773882 22806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:04:12.773929 22806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:12.773988 22806 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:04:12.774014 22806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:12.782286 22806 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:40889
I20250109 21:04:12.782306 22933 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:40889 every 8 connection(s)
I20250109 21:04:12.783280 22806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:04:12.790436 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:12.784887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:52535 (local address 127.12.126.126:41405)
0109 21:04:12.785006 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:12.785009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:12.785904 (+ 895us) server_negotiation.cc:407] Connection header received
0109 21:04:12.786632 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:12.786634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:12.786685 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:12.786772 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:12.788188 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:12.788669 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:12.789194 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:12.789304 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:12.790045 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:04:12.790063 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:04:12.790065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:04:12.790078 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:04:12.790097 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:04:12.790104 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:04:12.790159 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:12.790270 (+ 111us) server_negotiation.cc:299] Negotiation successful
0109 21:04:12.790321 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20250109 21:04:12.791160 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:12.785159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:41405 (local address 127.12.126.66:52535)
0109 21:04:12.785770 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:12.785802 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:12.786455 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:12.787022 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:12.787030 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:12.787117 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:12.788073 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:12.788088 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:12.788757 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:12.788759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:12.789123 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:12.789128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:12.789270 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:12.789839 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:04:12.789841 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:12.789871 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:04:12.789924 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:04:12.790171 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:04:12.790175 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:04:12.790179 (+ 4us) client_negotiation.cc:769] Sending connection context
0109 21:04:12.790270 (+ 91us) client_negotiation.cc:240] Negotiation successful
0109 21:04:12.790446 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20250109 21:04:12.791914 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 22806
I20250109 21:04:12.792042 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:04:12.792291 22934 heartbeater.cc:346] Connected to a master server at 127.12.126.126:41405
I20250109 21:04:12.792330 12793 external_mini_cluster.cc:1453] Setting key 51b748459d3ba4170bad8d096b511971
I20250109 21:04:12.792510 22934 heartbeater.cc:463] Registering TS with master...
I20250109 21:04:12.793073 22934 heartbeater.cc:510] Master 127.12.126.126:41405 requested a full tablet report, sending...
I20250109 21:04:12.794004 22615 ts_manager.cc:194] Registered new tserver with Master: c3e8fbf842f24253a2a07749f55ea250 (127.12.126.66:40889)
I20250109 21:04:12.794145 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:41405
--builtin_ntp_servers=127.12.126.84:42311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20250109 21:04:12.794593 22615 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.126.66:52535
W20250109 21:04:12.903906 22937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:04:12.904127 22937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:04:12.904172 22937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:04:12.907574 22937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:04:12.907639 22937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:04:12.907732 22937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:04:12.911756 22937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:41405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:04:12.912923 22937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:04:12.914083 22937 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:04:12.922508 22942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:12.923323 22943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:12.923980 22945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:04:12.924906 22937 server_base.cc:1034] running on GCE node
I20250109 21:04:12.925300 22937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:04:12.925956 22937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:04:12.927106 22937 hybrid_clock.cc:648] HybridClock initialized: now 1736456652927098 us; error 25 us; skew 500 ppm
I20250109 21:04:12.935309 22937 webserver.cc:458] Webserver started at http://127.12.126.67:38679/ using document root <none> and password file <none>
I20250109 21:04:12.936098 22937 fs_manager.cc:362] Metadata directory not provided
I20250109 21:04:12.936178 22937 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:04:12.936455 22937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:04:12.938445 22937 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "5296301e63e34aa8ad3efc0f489043c6"
format_stamp: "Formatted at 2025-01-09 21:04:12 on dist-test-slave-g0zb"
server_key: "1a9cb05b6af2b0f93b431ca05438c627"
server_key_iv: "dc630262cc3d41be3e7f67fa7b1aae3a"
server_key_version: "encryptionkey@0"
I20250109 21:04:12.938985 22937 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "5296301e63e34aa8ad3efc0f489043c6"
format_stamp: "Formatted at 2025-01-09 21:04:12 on dist-test-slave-g0zb"
server_key: "1a9cb05b6af2b0f93b431ca05438c627"
server_key_iv: "dc630262cc3d41be3e7f67fa7b1aae3a"
server_key_version: "encryptionkey@0"
I20250109 21:04:12.942483 22937 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:04:12.944779 22951 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:12.945837 22937 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:04:12.945966 22937 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "5296301e63e34aa8ad3efc0f489043c6"
format_stamp: "Formatted at 2025-01-09 21:04:12 on dist-test-slave-g0zb"
server_key: "1a9cb05b6af2b0f93b431ca05438c627"
server_key_iv: "dc630262cc3d41be3e7f67fa7b1aae3a"
server_key_version: "encryptionkey@0"
I20250109 21:04:12.946066 22937 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:04:12.981586 22937 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:04:12.982283 22937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:04:12.982458 22937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:04:12.983042 22937 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:04:12.983935 22937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:04:12.983983 22937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:12.984040 22937 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:04:12.984071 22937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:12.992611 22937 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:38731
I20250109 21:04:12.992641 23064 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:38731 every 8 connection(s)
I20250109 21:04:12.993557 22937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:04:13.000625 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 22937
I20250109 21:04:13.000725 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:04:13.000970 12793 external_mini_cluster.cc:1453] Setting key 30b69a7140d89ad31169368a7e12ec0d
I20250109 21:04:13.001885 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:12.995318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:45593 (local address 127.12.126.126:41405)
0109 21:04:12.995488 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:12.995493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:12.996629 (+ 1136us) server_negotiation.cc:407] Connection header received
0109 21:04:12.997461 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:12.997465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:12.997530 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:12.997610 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:12.999116 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:12.999627 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:13.000265 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:13.000440 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:13.001334 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:04:13.001355 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:04:13.001361 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:04:13.001378 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:04:13.001403 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:04:13.001412 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:04:13.001485 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:13.001673 (+ 188us) server_negotiation.cc:299] Negotiation successful
0109 21:04:13.001741 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250109 21:04:13.002705 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:12.995692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:41405 (local address 127.12.126.67:45593)
0109 21:04:12.996468 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:12.996525 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:12.997256 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:12.997798 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:12.997810 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:12.997886 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:12.998965 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:12.998982 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:12.999750 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:12.999753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:13.000141 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:13.000148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:13.000348 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:13.001089 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:04:13.001093 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:13.001139 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:04:13.001199 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:04:13.001505 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:04:13.001514 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:04:13.001523 (+ 9us) client_negotiation.cc:769] Sending connection context
0109 21:04:13.001678 (+ 155us) client_negotiation.cc:240] Negotiation successful
0109 21:04:13.001909 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":123,"threads_started":1}
I20250109 21:04:13.003801 23065 heartbeater.cc:346] Connected to a master server at 127.12.126.126:41405
I20250109 21:04:13.004002 23065 heartbeater.cc:463] Registering TS with master...
I20250109 21:04:13.004518 23065 heartbeater.cc:510] Master 127.12.126.126:41405 requested a full tablet report, sending...
I20250109 21:04:13.005493 22615 ts_manager.cc:194] Registered new tserver with Master: 5296301e63e34aa8ad3efc0f489043c6 (127.12.126.67:38731)
I20250109 21:04:13.006033 22615 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.126.67:45593
I20250109 21:04:13.015244 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250109 21:04:13.018023 22654 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:13.017159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35542 (local address 127.12.126.126:41405)
0109 21:04:13.017329 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:13.017333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:13.017422 (+ 89us) server_negotiation.cc:407] Connection header received
0109 21:04:13.017568 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:13.017571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:13.017619 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:13.017686 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:13.017915 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35542: BlockingRecv error: recv got EOF from 127.0.0.1:35542 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20250109 21:04:13.018276 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 22675
I20250109 21:04:13.024104 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 22806
I20250109 21:04:13.030484 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 22937
I20250109 21:04:13.036501 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 22584
2025-01-09T21:04:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (864 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2025-01-09T21:04:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:04:13Z Disabled control of system clock
I20250109 21:04:13.065575 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:38063
--webserver_interface=127.12.126.126
--webserver_port=0
--builtin_ntp_servers=127.12.126.84:39541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:38063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20250109 21:04:13.173646 23076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:04:13.173910 23076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:04:13.173959 23076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:04:13.177160 23076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:04:13.177230 23076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:04:13.177246 23076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:04:13.177266 23076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:04:13.177289 23076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:04:13.181138 23076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:38063
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:38063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:04:13.182282 23076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:04:13.183439 23076 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:04:13.191020 23082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:13.191130 23084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:13.191936 23081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:04:13.192636 23076 server_base.cc:1034] running on GCE node
I20250109 21:04:13.193145 23076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:04:13.194124 23076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:04:13.195286 23076 hybrid_clock.cc:648] HybridClock initialized: now 1736456653195268 us; error 30 us; skew 500 ppm
I20250109 21:04:13.197254 23076 webserver.cc:458] Webserver started at http://127.12.126.126:42429/ using document root <none> and password file <none>
I20250109 21:04:13.197851 23076 fs_manager.cc:362] Metadata directory not provided
I20250109 21:04:13.197913 23076 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:04:13.198161 23076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:04:13.200084 23076 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "fa4da046e91c4d0f9153c6b7bfc81d36"
format_stamp: "Formatted at 2025-01-09 21:04:13 on dist-test-slave-g0zb"
server_key: "1429d4175b4215321003f902f3bef7cc"
server_key_iv: "baae9debed9b3f4f2136a72b11788f93"
server_key_version: "encryptionkey@0"
I20250109 21:04:13.200577 23076 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "fa4da046e91c4d0f9153c6b7bfc81d36"
format_stamp: "Formatted at 2025-01-09 21:04:13 on dist-test-slave-g0zb"
server_key: "1429d4175b4215321003f902f3bef7cc"
server_key_iv: "baae9debed9b3f4f2136a72b11788f93"
server_key_version: "encryptionkey@0"
I20250109 21:04:13.203984 23076 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:04:13.206131 23090 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:13.207150 23076 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:04:13.207262 23076 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "fa4da046e91c4d0f9153c6b7bfc81d36"
format_stamp: "Formatted at 2025-01-09 21:04:13 on dist-test-slave-g0zb"
server_key: "1429d4175b4215321003f902f3bef7cc"
server_key_iv: "baae9debed9b3f4f2136a72b11788f93"
server_key_version: "encryptionkey@0"
I20250109 21:04:13.207351 23076 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:04:13.212282 23076 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:04:13.212860 23076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:04:13.213014 23076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:04:13.220443 23076 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:38063
I20250109 21:04:13.220458 23142 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:38063 every 8 connection(s)
I20250109 21:04:13.221971 23076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:04:13.224972 23143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:04:13.230736 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 23076
I20250109 21:04:13.230896 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:04:13.231060 23143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36: Bootstrap starting.
I20250109 21:04:13.231190 12793 external_mini_cluster.cc:1453] Setting key 3e03fe3d71683f183a29d328d994dde6
I20250109 21:04:13.233465 23143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36: Neither blocks nor log segments found. Creating new log.
I20250109 21:04:13.234180 23143 log.cc:826] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36: Log is configured to *not* fsync() on all Append() calls
I20250109 21:04:13.236038 23143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36: No bootstrap required, opened a new log
I20250109 21:04:13.237771 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:13.232438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.12.126.126:38063)
0109 21:04:13.233123 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:13.233136 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:13.233180 (+ 44us) server_negotiation.cc:407] Connection header received
0109 21:04:13.233805 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:13.233827 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:13.234141 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:13.234492 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:13.236484 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:04:13.236490 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:04:13.236502 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:04:13.236522 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:04:13.236584 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:04:13.236599 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:04:13.236720 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:13.236798 (+ 78us) server_negotiation.cc:299] Negotiation successful
0109 21:04:13.237031 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
I20250109 21:04:13.239691 23143 raft_consensus.cc:357] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4da046e91c4d0f9153c6b7bfc81d36" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38063 } }
I20250109 21:04:13.239928 23143 raft_consensus.cc:383] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:04:13.239998 23143 raft_consensus.cc:738] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4da046e91c4d0f9153c6b7bfc81d36, State: Initialized, Role: FOLLOWER
I20250109 21:04:13.240407 23143 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4da046e91c4d0f9153c6b7bfc81d36" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38063 } }
I20250109 21:04:13.240530 23143 raft_consensus.cc:397] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:04:13.240594 23143 raft_consensus.cc:491] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:04:13.240700 23143 raft_consensus.cc:3054] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:04:13.241539 23143 raft_consensus.cc:513] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4da046e91c4d0f9153c6b7bfc81d36" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38063 } }
I20250109 21:04:13.241853 23143 leader_election.cc:304] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa4da046e91c4d0f9153c6b7bfc81d36; no voters:
I20250109 21:04:13.242128 23143 leader_election.cc:290] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:04:13.242249 23148 raft_consensus.cc:2798] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:04:13.242460 23148 raft_consensus.cc:695] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36 [term 1 LEADER]: Becoming Leader. State: Replica: fa4da046e91c4d0f9153c6b7bfc81d36, State: Running, Role: LEADER
I20250109 21:04:13.242849 23148 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4da046e91c4d0f9153c6b7bfc81d36" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38063 } }
I20250109 21:04:13.243268 23143 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:04:13.244345 23149 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa4da046e91c4d0f9153c6b7bfc81d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4da046e91c4d0f9153c6b7bfc81d36" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38063 } } }
I20250109 21:04:13.244479 23149 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36 [sys.catalog]: This master's current role is: LEADER
I20250109 21:04:13.244757 23150 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa4da046e91c4d0f9153c6b7bfc81d36. Latest consensus state: current_term: 1 leader_uuid: "fa4da046e91c4d0f9153c6b7bfc81d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4da046e91c4d0f9153c6b7bfc81d36" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38063 } } }
I20250109 21:04:13.244850 23150 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36 [sys.catalog]: This master's current role is: LEADER
I20250109 21:04:13.245194 23157 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:04:13.248237 23157 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:04:13.253865 23157 catalog_manager.cc:1348] Generated new cluster ID: 40068479423242728be4094905f2be75
I20250109 21:04:13.253939 23157 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:04:13.268606 23157 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:04:13.268919 23157 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:04:13.277365 23157 catalog_manager.cc:5899] T 00000000000000000000000000000000 P fa4da046e91c4d0f9153c6b7bfc81d36: Generated new TSK 0
I20250109 21:04:13.278061 23157 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:04:13.285152 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:38063
--builtin_ntp_servers=127.12.126.84:39541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20250109 21:04:13.395089 23167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:04:13.395332 23167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:04:13.395407 23167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:04:13.398593 23167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:04:13.398654 23167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:04:13.398756 23167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:04:13.402537 23167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:38063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:04:13.403674 23167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:04:13.404836 23167 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:04:13.413159 23173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:13.414038 23175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:13.414729 23172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:04:13.415481 23167 server_base.cc:1034] running on GCE node
I20250109 21:04:13.415863 23167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:04:13.416419 23167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:04:13.417580 23167 hybrid_clock.cc:648] HybridClock initialized: now 1736456653417558 us; error 35 us; skew 500 ppm
I20250109 21:04:13.419582 23167 webserver.cc:458] Webserver started at http://127.12.126.65:35649/ using document root <none> and password file <none>
I20250109 21:04:13.420158 23167 fs_manager.cc:362] Metadata directory not provided
I20250109 21:04:13.420213 23167 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:04:13.420408 23167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:04:13.422201 23167 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "e7ae07e146804a9ea07465d4e18b7d70"
format_stamp: "Formatted at 2025-01-09 21:04:13 on dist-test-slave-g0zb"
server_key: "befec552ce8ae4f81be19f2f494c6676"
server_key_iv: "93abdcbc5edc2cdebbb7a3b11dd947e2"
server_key_version: "encryptionkey@0"
I20250109 21:04:13.422739 23167 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "e7ae07e146804a9ea07465d4e18b7d70"
format_stamp: "Formatted at 2025-01-09 21:04:13 on dist-test-slave-g0zb"
server_key: "befec552ce8ae4f81be19f2f494c6676"
server_key_iv: "93abdcbc5edc2cdebbb7a3b11dd947e2"
server_key_version: "encryptionkey@0"
I20250109 21:04:13.426203 23167 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250109 21:04:13.428277 23181 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:13.429287 23167 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:04:13.429400 23167 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "e7ae07e146804a9ea07465d4e18b7d70"
format_stamp: "Formatted at 2025-01-09 21:04:13 on dist-test-slave-g0zb"
server_key: "befec552ce8ae4f81be19f2f494c6676"
server_key_iv: "93abdcbc5edc2cdebbb7a3b11dd947e2"
server_key_version: "encryptionkey@0"
I20250109 21:04:13.429488 23167 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:04:13.434435 23167 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:04:13.435007 23167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:04:13.435158 23167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:04:13.435635 23167 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:04:13.436481 23167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:04:13.436529 23167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:13.436594 23167 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:04:13.436619 23167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:13.445214 23167 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:46337
I20250109 21:04:13.445340 23294 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:46337 every 8 connection(s)
I20250109 21:04:13.446269 23167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:04:13.450975 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 23167
I20250109 21:04:13.451130 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:04:13.451404 12793 external_mini_cluster.cc:1453] Setting key 94d4ef78e4a0ced231cbb50563664c5c
I20250109 21:04:13.452044 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:13.447922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:44967 (local address 127.12.126.126:38063)
0109 21:04:13.448090 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:13.448095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:13.448974 (+ 879us) server_negotiation.cc:407] Connection header received
0109 21:04:13.449735 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:13.449738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:13.449779 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:13.449886 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:13.451596 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:04:13.451601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:04:13.451604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:04:13.451617 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:04:13.451638 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:04:13.451645 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:04:13.451730 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:13.451862 (+ 132us) server_negotiation.cc:299] Negotiation successful
0109 21:04:13.451909 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20250109 21:04:13.452798 23297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:13.448246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:38063 (local address 127.12.126.65:44967)
0109 21:04:13.448832 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:13.448865 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:13.449530 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:13.450028 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:13.450036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:13.450119 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:13.451312 (+ 1193us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:04:13.451314 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:13.451350 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:04:13.451412 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:04:13.451719 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:04:13.451726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:04:13.451732 (+ 6us) client_negotiation.cc:769] Sending connection context
0109 21:04:13.451921 (+ 189us) client_negotiation.cc:240] Negotiation successful
0109 21:04:13.452142 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
I20250109 21:04:13.453349 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:38063
--builtin_ntp_servers=127.12.126.84:39541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20250109 21:04:13.454027 23295 heartbeater.cc:346] Connected to a master server at 127.12.126.126:38063
I20250109 21:04:13.454241 23295 heartbeater.cc:463] Registering TS with master...
I20250109 21:04:13.454620 23295 heartbeater.cc:510] Master 127.12.126.126:38063 requested a full tablet report, sending...
I20250109 21:04:13.456053 23107 ts_manager.cc:194] Registered new tserver with Master: e7ae07e146804a9ea07465d4e18b7d70 (127.12.126.65:46337)
W20250109 21:04:13.577308 23298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:04:13.577548 23298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:04:13.577630 23298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:04:13.580718 23298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:04:13.580780 23298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:04:13.580891 23298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:04:13.584636 23298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:38063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:04:13.585747 23298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:04:13.586933 23298 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:04:13.595026 23306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:13.595706 23303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:13.596311 23304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:04:13.597122 23298 server_base.cc:1034] running on GCE node
I20250109 21:04:13.597489 23298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:04:13.598043 23298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:04:13.599186 23298 hybrid_clock.cc:648] HybridClock initialized: now 1736456653599153 us; error 45 us; skew 500 ppm
I20250109 21:04:13.601179 23298 webserver.cc:458] Webserver started at http://127.12.126.66:42037/ using document root <none> and password file <none>
I20250109 21:04:13.601773 23298 fs_manager.cc:362] Metadata directory not provided
I20250109 21:04:13.601828 23298 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:04:13.602020 23298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:04:13.603931 23298 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "0a9427051b3d4a68ac90bbe66d3e7e00"
format_stamp: "Formatted at 2025-01-09 21:04:13 on dist-test-slave-g0zb"
server_key: "8923253e480d3efba638c6ec54501058"
server_key_iv: "ede64c7b3cd895d7918f6ad2355fd04e"
server_key_version: "encryptionkey@0"
I20250109 21:04:13.604427 23298 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "0a9427051b3d4a68ac90bbe66d3e7e00"
format_stamp: "Formatted at 2025-01-09 21:04:13 on dist-test-slave-g0zb"
server_key: "8923253e480d3efba638c6ec54501058"
server_key_iv: "ede64c7b3cd895d7918f6ad2355fd04e"
server_key_version: "encryptionkey@0"
I20250109 21:04:13.607952 23298 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:04:13.610188 23312 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:13.611374 23298 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:04:13.611496 23298 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "0a9427051b3d4a68ac90bbe66d3e7e00"
format_stamp: "Formatted at 2025-01-09 21:04:13 on dist-test-slave-g0zb"
server_key: "8923253e480d3efba638c6ec54501058"
server_key_iv: "ede64c7b3cd895d7918f6ad2355fd04e"
server_key_version: "encryptionkey@0"
I20250109 21:04:13.611603 23298 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:04:13.616647 23298 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:04:13.617183 23298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:04:13.617367 23298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:04:13.617866 23298 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:04:13.618734 23298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:04:13.618784 23298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:13.618906 23298 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:04:13.618940 23298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:13.627666 23298 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:34983
I20250109 21:04:13.627689 23425 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:34983 every 8 connection(s)
I20250109 21:04:13.628625 23298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:04:13.634303 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:13.630322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:54155 (local address 127.12.126.126:38063)
0109 21:04:13.630468 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:13.630471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:13.631372 (+ 901us) server_negotiation.cc:407] Connection header received
0109 21:04:13.632134 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:13.632138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:13.632187 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:13.632274 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:13.633849 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:04:13.633853 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:04:13.633856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:04:13.633870 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:04:13.633891 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:04:13.633898 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:04:13.633984 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:13.634114 (+ 130us) server_negotiation.cc:299] Negotiation successful
0109 21:04:13.634162 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250109 21:04:13.635037 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:13.630624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:38063 (local address 127.12.126.66:54155)
0109 21:04:13.631238 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:13.631272 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:13.631915 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:13.632436 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:13.632444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:13.632524 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:13.633539 (+ 1015us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:04:13.633541 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:13.633574 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:04:13.633631 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:04:13.633971 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:04:13.633978 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:04:13.633984 (+ 6us) client_negotiation.cc:769] Sending connection context
0109 21:04:13.634172 (+ 188us) client_negotiation.cc:240] Negotiation successful
0109 21:04:13.634397 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":111,"threads_started":1}
I20250109 21:04:13.636054 23426 heartbeater.cc:346] Connected to a master server at 127.12.126.126:38063
I20250109 21:04:13.636258 23426 heartbeater.cc:463] Registering TS with master...
I20250109 21:04:13.636641 23426 heartbeater.cc:510] Master 127.12.126.126:38063 requested a full tablet report, sending...
I20250109 21:04:13.637039 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 23298
I20250109 21:04:13.637149 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:04:13.637401 12793 external_mini_cluster.cc:1453] Setting key a3090f14622714d18c12ecc67e7a3a72
I20250109 21:04:13.637645 23107 ts_manager.cc:194] Registered new tserver with Master: 0a9427051b3d4a68ac90bbe66d3e7e00 (127.12.126.66:34983)
I20250109 21:04:13.639317 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:38063
--builtin_ntp_servers=127.12.126.84:39541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20250109 21:04:13.748556 23429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:04:13.748796 23429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:04:13.748849 23429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:04:13.752112 23429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:04:13.752177 23429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:04:13.752280 23429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:04:13.756098 23429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:38063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:04:13.757185 23429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:04:13.758298 23429 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:04:13.766418 23434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:13.767066 23435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:13.767782 23437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:04:13.768615 23429 server_base.cc:1034] running on GCE node
I20250109 21:04:13.768975 23429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:04:13.769524 23429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:04:13.770669 23429 hybrid_clock.cc:648] HybridClock initialized: now 1736456653770659 us; error 26 us; skew 500 ppm
I20250109 21:04:13.772606 23429 webserver.cc:458] Webserver started at http://127.12.126.67:42345/ using document root <none> and password file <none>
I20250109 21:04:13.773180 23429 fs_manager.cc:362] Metadata directory not provided
I20250109 21:04:13.773239 23429 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:04:13.773432 23429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:04:13.775316 23429 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "a833c0dadd7740949067a8038d8b4f99"
format_stamp: "Formatted at 2025-01-09 21:04:13 on dist-test-slave-g0zb"
server_key: "80b38bf8359c3cda975dbbd039cb707b"
server_key_iv: "e4cdf9d4d61d0d169412c1437e7ce17f"
server_key_version: "encryptionkey@0"
I20250109 21:04:13.775810 23429 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "a833c0dadd7740949067a8038d8b4f99"
format_stamp: "Formatted at 2025-01-09 21:04:13 on dist-test-slave-g0zb"
server_key: "80b38bf8359c3cda975dbbd039cb707b"
server_key_iv: "e4cdf9d4d61d0d169412c1437e7ce17f"
server_key_version: "encryptionkey@0"
I20250109 21:04:13.779310 23429 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:04:13.781451 23443 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:13.782446 23429 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:04:13.782553 23429 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "a833c0dadd7740949067a8038d8b4f99"
format_stamp: "Formatted at 2025-01-09 21:04:13 on dist-test-slave-g0zb"
server_key: "80b38bf8359c3cda975dbbd039cb707b"
server_key_iv: "e4cdf9d4d61d0d169412c1437e7ce17f"
server_key_version: "encryptionkey@0"
I20250109 21:04:13.782651 23429 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:04:13.787437 23429 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:04:13.787953 23429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:04:13.788102 23429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:04:13.788611 23429 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:04:13.789474 23429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:04:13.789520 23429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:13.789559 23429 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:04:13.789582 23429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:13.797947 23429 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:34775
I20250109 21:04:13.797984 23556 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:34775 every 8 connection(s)
I20250109 21:04:13.798940 23429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:04:13.804643 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:13.800638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:35111 (local address 127.12.126.126:38063)
0109 21:04:13.800776 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:13.800779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:13.801710 (+ 931us) server_negotiation.cc:407] Connection header received
0109 21:04:13.802498 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:13.802502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:13.802550 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:13.802635 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:13.804156 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:04:13.804160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:04:13.804163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:04:13.804176 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:04:13.804197 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:04:13.804204 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:04:13.804297 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:13.804451 (+ 154us) server_negotiation.cc:299] Negotiation successful
0109 21:04:13.804501 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250109 21:04:13.804929 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 23429
I20250109 21:04:13.805048 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:04:13.805274 12793 external_mini_cluster.cc:1453] Setting key aa99a1d21fb616f0bd7791fa13e15a51
I20250109 21:04:13.805413 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:13.800926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:38063 (local address 127.12.126.67:35111)
0109 21:04:13.801565 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:13.801616 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:13.802297 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:13.802791 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:13.802818 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:13.802922 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:13.803909 (+ 987us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:04:13.803910 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:13.803942 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:04:13.804000 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:04:13.804276 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:04:13.804283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:04:13.804288 (+ 5us) client_negotiation.cc:769] Sending connection context
0109 21:04:13.804505 (+ 217us) client_negotiation.cc:240] Negotiation successful
0109 21:04:13.804733 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
I20250109 21:04:13.806502 23557 heartbeater.cc:346] Connected to a master server at 127.12.126.126:38063
I20250109 21:04:13.806715 23557 heartbeater.cc:463] Registering TS with master...
I20250109 21:04:13.807183 23557 heartbeater.cc:510] Master 127.12.126.126:38063 requested a full tablet report, sending...
I20250109 21:04:13.808121 23106 ts_manager.cc:194] Registered new tserver with Master: a833c0dadd7740949067a8038d8b4f99 (127.12.126.67:34775)
I20250109 21:04:13.819136 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250109 21:04:13.821696 23146 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:04:13.820942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54708 (local address 127.12.126.126:38063)
0109 21:04:13.821103 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:13.821107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:13.821191 (+ 84us) server_negotiation.cc:407] Connection header received
0109 21:04:13.821297 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:13.821299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:13.821335 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:13.821394 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:13.821565 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54708: BlockingRecv error: recv got EOF from 127.0.0.1:54708 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20250109 21:04:13.821795 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 23167
I20250109 21:04:13.827586 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 23298
I20250109 21:04:13.832818 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 23429
I20250109 21:04:13.838076 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 23076
2025-01-09T21:04:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (800 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:04:13 dist-test-slave-g0zb krb5kdc[23567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:04:13 dist-test-slave-g0zb krb5kdc[23567](info): set up 2 sockets
Jan 09 21:04:13 dist-test-slave-g0zb krb5kdc[23567](info): commencing operation
krb5kdc: starting...
W20250109 21:04:14.832160 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:04:14 dist-test-slave-g0zb krb5kdc[23567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:04:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:04:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:04:14.949750 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:42055
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:42055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:04:15.055724 23580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:04:15.055979 23580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:04:15.056046 23580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:04:15.059549 23580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:04:15.059617 23580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:04:15.059639 23580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:04:15.059662 23580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:04:15.059700 23580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:04:15.064024 23580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:42055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:42055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:04:15.065081 23580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:04:15.065923 23580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:04:15.073562 23588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:15.073688 23586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:15.074513 23585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:04:15.075297 23580 server_base.cc:1034] running on GCE node
I20250109 21:04:15.075827 23580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:04:15.076783 23580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:04:15.077934 23580 hybrid_clock.cc:648] HybridClock initialized: now 1736456655077913 us; error 38 us; skew 500 ppm
Jan 09 21:04:15 dist-test-slave-g0zb krb5kdc[23567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456655, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:04:15.080538 23580 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:04:15.081511 23580 webserver.cc:458] Webserver started at http://127.12.126.126:36147/ using document root <none> and password file <none>
I20250109 21:04:15.082057 23580 fs_manager.cc:362] Metadata directory not provided
I20250109 21:04:15.082116 23580 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:04:15.082305 23580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:04:15.084136 23580 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "d7099a8594a74df9bad55509317089ea"
format_stamp: "Formatted at 2025-01-09 21:04:15 on dist-test-slave-g0zb"
server_key: "0741db2cfbc887b6e8a36670b955ded5"
server_key_iv: "d6609b49ac2e8fb1e19e50972a37bd0c"
server_key_version: "encryptionkey@0"
I20250109 21:04:15.084618 23580 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "d7099a8594a74df9bad55509317089ea"
format_stamp: "Formatted at 2025-01-09 21:04:15 on dist-test-slave-g0zb"
server_key: "0741db2cfbc887b6e8a36670b955ded5"
server_key_iv: "d6609b49ac2e8fb1e19e50972a37bd0c"
server_key_version: "encryptionkey@0"
I20250109 21:04:15.088008 23580 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:04:15.090022 23595 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:15.091053 23580 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:04:15.091164 23580 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "d7099a8594a74df9bad55509317089ea"
format_stamp: "Formatted at 2025-01-09 21:04:15 on dist-test-slave-g0zb"
server_key: "0741db2cfbc887b6e8a36670b955ded5"
server_key_iv: "d6609b49ac2e8fb1e19e50972a37bd0c"
server_key_version: "encryptionkey@0"
I20250109 21:04:15.091264 23580 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:04:15.107714 23580 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:04:15.110402 23580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:04:15.110586 23580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:04:15.117040 23580 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:42055
I20250109 21:04:15.117051 23647 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:42055 every 8 connection(s)
I20250109 21:04:15.117980 23580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:04:15.120805 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:04:15.125325 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 23580
I20250109 21:04:15.125483 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:04:15.125735 12793 external_mini_cluster.cc:1453] Setting key 2d6bf106d1e2ad9cc2894c5a937ff4ff
I20250109 21:04:15.126019 23648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea: Bootstrap starting.
I20250109 21:04:15.128279 23648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea: Neither blocks nor log segments found. Creating new log.
I20250109 21:04:15.128938 23648 log.cc:826] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea: Log is configured to *not* fsync() on all Append() calls
I20250109 21:04:15.130976 23648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea: No bootstrap required, opened a new log
Jan 09 21:04:15 dist-test-slave-g0zb krb5kdc[23567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:04:15.134409 23648 raft_consensus.cc:357] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7099a8594a74df9bad55509317089ea" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42055 } }
I20250109 21:04:15.134588 23648 raft_consensus.cc:383] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:04:15.134626 23648 raft_consensus.cc:738] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7099a8594a74df9bad55509317089ea, State: Initialized, Role: FOLLOWER
I20250109 21:04:15.135076 23648 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7099a8594a74df9bad55509317089ea" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42055 } }
I20250109 21:04:15.135190 23648 raft_consensus.cc:397] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:04:15.135248 23648 raft_consensus.cc:491] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:04:15.135360 23648 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:04:15.136198 23648 raft_consensus.cc:513] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7099a8594a74df9bad55509317089ea" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42055 } }
I20250109 21:04:15.136574 23648 leader_election.cc:304] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7099a8594a74df9bad55509317089ea; no voters:
I20250109 21:04:15.136910 23648 leader_election.cc:290] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:04:15.136997 23653 raft_consensus.cc:2798] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:04:15.137187 23653 raft_consensus.cc:695] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea [term 1 LEADER]: Becoming Leader. State: Replica: d7099a8594a74df9bad55509317089ea, State: Running, Role: LEADER
I20250109 21:04:15.137526 23653 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7099a8594a74df9bad55509317089ea" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42055 } }
I20250109 21:04:15.138140 23648 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:04:15.138868 23653 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7099a8594a74df9bad55509317089ea. Latest consensus state: current_term: 1 leader_uuid: "d7099a8594a74df9bad55509317089ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7099a8594a74df9bad55509317089ea" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42055 } } }
I20250109 21:04:15.139015 23653 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea [sys.catalog]: This master's current role is: LEADER
I20250109 21:04:15.139364 23654 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7099a8594a74df9bad55509317089ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7099a8594a74df9bad55509317089ea" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42055 } } }
I20250109 21:04:15.139529 23654 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea [sys.catalog]: This master's current role is: LEADER
I20250109 21:04:15.139600 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:15.126951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35806 (local address 127.12.126.126:42055)
0109 21:04:15.127380 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:15.127389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:15.127422 (+ 33us) server_negotiation.cc:407] Connection header received
0109 21:04:15.127938 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:15.127955 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:15.128227 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:15.128570 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:15.129498 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.130682 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:15.131398 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.131643 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:15.133968 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:04:15.133987 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:04:15.133999 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:04:15.134033 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:04:15.135897 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:04:15.136439 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:04:15.136444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:04:15.136449 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:04:15.136523 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:04:15.136769 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:04:15.136771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:04:15.136773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:04:15.137242 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:04:15.138092 (+ 850us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:15.138261 (+ 169us) server_negotiation.cc:299] Negotiation successful
0109 21:04:15.138545 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":143,"threads_started":1}
W20250109 21:04:15.142980 23668 catalog_manager.cc:1559] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:04:15.143049 23668 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 21:04:15.143115 23661 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:04:15.143812 23661 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:04:15.149006 23661 catalog_manager.cc:1348] Generated new cluster ID: 8faaccd25e5347fdb53ae930fa830305
I20250109 21:04:15.149080 23661 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:04:15.157020 23661 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:04:15.157868 23661 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:04:15.163522 23661 catalog_manager.cc:5899] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea: Generated new TSK 0
I20250109 21:04:15.164170 23661 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
I20250109 21:04:15.194299 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42055
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:04:15.314338 23674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:04:15.314605 23674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:04:15.314663 23674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:04:15.317730 23674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:04:15.317798 23674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:04:15.317891 23674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:04:15.322116 23674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:04:15.323266 23674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:04:15.324122 23674 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:04:15.332461 23679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:15.332715 23682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:15.333441 23680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:04:15.334167 23674 server_base.cc:1034] running on GCE node
I20250109 21:04:15.334535 23674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:04:15.335127 23674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:04:15.336265 23674 hybrid_clock.cc:648] HybridClock initialized: now 1736456655336257 us; error 42 us; skew 500 ppm
Jan 09 21:04:15 dist-test-slave-g0zb krb5kdc[23567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456655, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:04:15.338964 23674 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:04:15.339906 23674 webserver.cc:458] Webserver started at http://127.12.126.65:35087/ using document root <none> and password file <none>
I20250109 21:04:15.340515 23674 fs_manager.cc:362] Metadata directory not provided
I20250109 21:04:15.340588 23674 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:04:15.340785 23674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:04:15.342613 23674 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "05493057fa3b4f439fa5089a7d4fbf54"
format_stamp: "Formatted at 2025-01-09 21:04:15 on dist-test-slave-g0zb"
server_key: "64e9fa6d6211329631cea91bf70f301e"
server_key_iv: "f5d19384d7b751b1f7ad90440aa64ec8"
server_key_version: "encryptionkey@0"
I20250109 21:04:15.343107 23674 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "05493057fa3b4f439fa5089a7d4fbf54"
format_stamp: "Formatted at 2025-01-09 21:04:15 on dist-test-slave-g0zb"
server_key: "64e9fa6d6211329631cea91bf70f301e"
server_key_iv: "f5d19384d7b751b1f7ad90440aa64ec8"
server_key_version: "encryptionkey@0"
I20250109 21:04:15.346410 23674 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250109 21:04:15.348604 23689 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:15.349630 23674 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:04:15.349741 23674 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "05493057fa3b4f439fa5089a7d4fbf54"
format_stamp: "Formatted at 2025-01-09 21:04:15 on dist-test-slave-g0zb"
server_key: "64e9fa6d6211329631cea91bf70f301e"
server_key_iv: "f5d19384d7b751b1f7ad90440aa64ec8"
server_key_version: "encryptionkey@0"
I20250109 21:04:15.349830 23674 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:04:15.367341 23674 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:04:15.370385 23674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:04:15.370576 23674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:04:15.371155 23674 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:04:15.371942 23674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:04:15.371991 23674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:15.372049 23674 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:04:15.372079 23674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:15.380663 23674 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:33103
I20250109 21:04:15.380697 23802 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:33103 every 8 connection(s)
I20250109 21:04:15.381655 23674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
Jan 09 21:04:15 dist-test-slave-g0zb krb5kdc[23567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456655, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:04:15.391057 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 23674
I20250109 21:04:15.391163 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:04:15.391425 12793 external_mini_cluster.cc:1453] Setting key 4ec3d047483b18bc1be48331dd251a34
I20250109 21:04:15.394320 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:15.383449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:40197 (local address 127.12.126.126:42055)
0109 21:04:15.383595 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:15.383598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:15.384729 (+ 1131us) server_negotiation.cc:407] Connection header received
0109 21:04:15.385566 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:15.385570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:15.385633 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:15.385711 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:15.387046 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.387546 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:15.388165 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.388341 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:15.390549 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:04:15.390577 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:04:15.390579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:04:15.390604 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:04:15.392217 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:04:15.392799 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:04:15.392803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:04:15.392805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:04:15.392851 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:04:15.393134 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:04:15.393138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:04:15.393140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:04:15.393342 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:04:15.393451 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:15.393918 (+ 467us) server_negotiation.cc:299] Negotiation successful
0109 21:04:15.394068 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250109 21:04:15.394779 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:15.384162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42055 (local address 127.12.126.65:40197)
0109 21:04:15.384594 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:15.384629 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:15.385351 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:15.385882 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:15.385891 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:15.386272 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:15.386910 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:15.386924 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.387673 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:15.387677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:15.388059 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:15.388066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.388277 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:15.388935 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:15.388960 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:15.390397 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:04:15.392325 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:04:15.392331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:04:15.392343 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:04:15.392702 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:04:15.392950 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:04:15.392953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:04:15.392954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:04:15.393045 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:04:15.393445 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:04:15.393450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:04:15.393685 (+ 235us) client_negotiation.cc:769] Sending connection context
0109 21:04:15.393851 (+ 166us) client_negotiation.cc:240] Negotiation successful
0109 21:04:15.394069 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":114,"threads_started":1}
I20250109 21:04:15.395987 23803 heartbeater.cc:346] Connected to a master server at 127.12.126.126:42055
I20250109 21:04:15.396214 23803 heartbeater.cc:463] Registering TS with master...
I20250109 21:04:15.396709 23803 heartbeater.cc:510] Master 127.12.126.126:42055 requested a full tablet report, sending...
I20250109 21:04:15.398375 23612 ts_manager.cc:194] Registered new tserver with Master: 05493057fa3b4f439fa5089a7d4fbf54 (127.12.126.65:33103)
I20250109 21:04:15.400138 23612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:40197
WARNING: no policy specified for kudu/127.12.126.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
I20250109 21:04:15.418689 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42055
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:04:15.523942 23808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:04:15.524181 23808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:04:15.524247 23808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:04:15.527132 23808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:04:15.527187 23808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:04:15.527274 23808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:04:15.531378 23808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:04:15.532404 23808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:04:15.533218 23808 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:04:15.541697 23816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:15.542507 23813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:15.542707 23814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:04:15.543479 23808 server_base.cc:1034] running on GCE node
I20250109 21:04:15.543846 23808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:04:15.544405 23808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:04:15.545560 23808 hybrid_clock.cc:648] HybridClock initialized: now 1736456655545550 us; error 24 us; skew 500 ppm
Jan 09 21:04:15 dist-test-slave-g0zb krb5kdc[23567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456655, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:04:15.548153 23808 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:04:15.549000 23808 webserver.cc:458] Webserver started at http://127.12.126.66:36173/ using document root <none> and password file <none>
I20250109 21:04:15.549510 23808 fs_manager.cc:362] Metadata directory not provided
I20250109 21:04:15.549567 23808 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:04:15.549722 23808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:04:15.551558 23808 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "c196489edf9346c4bd5c29b020a4d5ba"
format_stamp: "Formatted at 2025-01-09 21:04:15 on dist-test-slave-g0zb"
server_key: "3645cbbd09953e5400868418fc6c062e"
server_key_iv: "231b71b338dc0f967dce71d5ed88e2fe"
server_key_version: "encryptionkey@0"
I20250109 21:04:15.551981 23808 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "c196489edf9346c4bd5c29b020a4d5ba"
format_stamp: "Formatted at 2025-01-09 21:04:15 on dist-test-slave-g0zb"
server_key: "3645cbbd09953e5400868418fc6c062e"
server_key_iv: "231b71b338dc0f967dce71d5ed88e2fe"
server_key_version: "encryptionkey@0"
I20250109 21:04:15.555378 23808 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:04:15.557531 23823 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:15.558497 23808 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:04:15.558599 23808 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "c196489edf9346c4bd5c29b020a4d5ba"
format_stamp: "Formatted at 2025-01-09 21:04:15 on dist-test-slave-g0zb"
server_key: "3645cbbd09953e5400868418fc6c062e"
server_key_iv: "231b71b338dc0f967dce71d5ed88e2fe"
server_key_version: "encryptionkey@0"
I20250109 21:04:15.558694 23808 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:04:15.568065 23808 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:04:15.570574 23808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:04:15.570712 23808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:04:15.571262 23808 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:04:15.571988 23808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:04:15.572032 23808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:15.572070 23808 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:04:15.572090 23808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:15.580363 23808 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:36707
I20250109 21:04:15.580381 23936 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:36707 every 8 connection(s)
I20250109 21:04:15.581364 23808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:04:15.584286 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 23808
I20250109 21:04:15.584419 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:04:15.584704 12793 external_mini_cluster.cc:1453] Setting key 1c6fe19723bf147e2aacae32d6462c04
Jan 09 21:04:15 dist-test-slave-g0zb krb5kdc[23567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456655, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.126.67@KRBTEST.COM; defaulting to no policy
I20250109 21:04:15.594973 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:15.583157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:47217 (local address 127.12.126.126:42055)
0109 21:04:15.583304 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:15.583308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:15.584577 (+ 1269us) server_negotiation.cc:407] Connection header received
0109 21:04:15.585501 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:15.585504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:15.585566 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:15.585643 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:15.587135 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.587913 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:15.588774 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.589001 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:15.591551 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:04:15.591569 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:04:15.591571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:04:15.591599 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:04:15.592977 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:04:15.593584 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:04:15.593586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:04:15.593588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:04:15.593624 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:04:15.593890 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:04:15.593891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:04:15.593893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:04:15.594070 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:04:15.594149 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:15.594666 (+ 517us) server_negotiation.cc:299] Negotiation successful
0109 21:04:15.594799 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250109 21:04:15.595721 23939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:15.583919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42055 (local address 127.12.126.66:47217)
0109 21:04:15.584413 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:15.584461 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:15.585293 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:15.585834 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:15.585844 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:15.586374 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:15.586982 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:15.586999 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.588045 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:15.588052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:15.588654 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:15.588665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.588848 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:15.589441 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:15.589466 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:15.591359 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:04:15.593102 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:04:15.593110 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:04:15.593125 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:04:15.593474 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:04:15.593700 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:04:15.593703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:04:15.593705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:04:15.593815 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:04:15.594174 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:04:15.594183 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:04:15.594454 (+ 271us) client_negotiation.cc:769] Sending connection context
0109 21:04:15.594638 (+ 184us) client_negotiation.cc:240] Negotiation successful
0109 21:04:15.594934 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":116,"threads_started":1}
I20250109 21:04:15.596810 23937 heartbeater.cc:346] Connected to a master server at 127.12.126.126:42055
I20250109 21:04:15.597060 23937 heartbeater.cc:463] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.67@KRBTEST.COM" created.
I20250109 21:04:15.597673 23937 heartbeater.cc:510] Master 127.12.126.126:42055 requested a full tablet report, sending...
I20250109 21:04:15.598716 23612 ts_manager.cc:194] Registered new tserver with Master: c196489edf9346c4bd5c29b020a4d5ba (127.12.126.66:36707)
I20250109 21:04:15.599323 23612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:47217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
I20250109 21:04:15.611887 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42055
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:04:15.718775 23942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:04:15.719069 23942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:04:15.719127 23942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:04:15.722263 23942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:04:15.722326 23942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:04:15.722401 23942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:04:15.726519 23942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:04:15.727591 23942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:04:15.728420 23942 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:04:15.736860 23950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:15.737073 23947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:04:15.737813 23948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:04:15.738655 23942 server_base.cc:1034] running on GCE node
I20250109 21:04:15.739104 23942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:04:15.739665 23942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:04:15.740810 23942 hybrid_clock.cc:648] HybridClock initialized: now 1736456655740799 us; error 25 us; skew 500 ppm
Jan 09 21:04:15 dist-test-slave-g0zb krb5kdc[23567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456655, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:04:15.743541 23942 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:04:15.744459 23942 webserver.cc:458] Webserver started at http://127.12.126.67:38817/ using document root <none> and password file <none>
I20250109 21:04:15.744998 23942 fs_manager.cc:362] Metadata directory not provided
I20250109 21:04:15.745047 23942 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:04:15.745203 23942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:04:15.747049 23942 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "b222cc54fdc449958c62721027cc07c3"
format_stamp: "Formatted at 2025-01-09 21:04:15 on dist-test-slave-g0zb"
server_key: "190fcdb1c24c63ae921b74d5f3765c8c"
server_key_iv: "ea6553a62ddb56455f433018b8571ee9"
server_key_version: "encryptionkey@0"
I20250109 21:04:15.747489 23942 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "b222cc54fdc449958c62721027cc07c3"
format_stamp: "Formatted at 2025-01-09 21:04:15 on dist-test-slave-g0zb"
server_key: "190fcdb1c24c63ae921b74d5f3765c8c"
server_key_iv: "ea6553a62ddb56455f433018b8571ee9"
server_key_version: "encryptionkey@0"
I20250109 21:04:15.750773 23942 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250109 21:04:15.752908 23957 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:15.753943 23942 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:04:15.754055 23942 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "b222cc54fdc449958c62721027cc07c3"
format_stamp: "Formatted at 2025-01-09 21:04:15 on dist-test-slave-g0zb"
server_key: "190fcdb1c24c63ae921b74d5f3765c8c"
server_key_iv: "ea6553a62ddb56455f433018b8571ee9"
server_key_version: "encryptionkey@0"
I20250109 21:04:15.754150 23942 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:04:15.788004 23942 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:04:15.790673 23942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:04:15.790848 23942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:04:15.791334 23942 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:04:15.792083 23942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:04:15.792140 23942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:15.792179 23942 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:04:15.792200 23942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:04:15.800820 23942 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:42353
I20250109 21:04:15.800850 24070 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:42353 every 8 connection(s)
I20250109 21:04:15.801772 23942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:04:15.808025 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 23942
I20250109 21:04:15.808161 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:04:15.808439 12793 external_mini_cluster.cc:1453] Setting key 3325e79be8664984b8315effd95c76a6
Jan 09 21:04:15 dist-test-slave-g0zb krb5kdc[23567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456655, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:04:15.813416 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:15.803550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:43215 (local address 127.12.126.126:42055)
0109 21:04:15.803705 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:15.803709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:15.804764 (+ 1055us) server_negotiation.cc:407] Connection header received
0109 21:04:15.805491 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:15.805494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:15.805544 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:15.805626 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:15.806838 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.807326 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:15.807938 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.808054 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:15.810294 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:04:15.810309 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:04:15.810311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:04:15.810337 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:04:15.811610 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:04:15.812060 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:04:15.812064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:04:15.812065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:04:15.812108 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:04:15.812381 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:04:15.812384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:04:15.812386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:04:15.812549 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:04:15.812639 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:15.813134 (+ 495us) server_negotiation.cc:299] Negotiation successful
0109 21:04:15.813252 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250109 21:04:15.814018 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:15.804221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42055 (local address 127.12.126.67:43215)
0109 21:04:15.804632 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:15.804665 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:15.805314 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:15.805764 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:15.805771 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:15.806133 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:04:15.806698 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:15.806710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.807417 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:15.807420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:15.807838 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:15.807849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.808070 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:15.808700 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:04:15.808734 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:04:15.810138 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:04:15.811715 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:04:15.811721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:04:15.811733 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:04:15.811965 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:04:15.812186 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:04:15.812190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:04:15.812193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:04:15.812280 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:04:15.812646 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:04:15.812654 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:04:15.812891 (+ 237us) client_negotiation.cc:769] Sending connection context
0109 21:04:15.813086 (+ 195us) client_negotiation.cc:240] Negotiation successful
0109 21:04:15.813362 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":94,"threads_started":1}
I20250109 21:04:15.814976 24071 heartbeater.cc:346] Connected to a master server at 127.12.126.126:42055
I20250109 21:04:15.815191 24071 heartbeater.cc:463] Registering TS with master...
I20250109 21:04:15.815704 24071 heartbeater.cc:510] Master 127.12.126.126:42055 requested a full tablet report, sending...
I20250109 21:04:15.816649 23612 ts_manager.cc:194] Registered new tserver with Master: b222cc54fdc449958c62721027cc07c3 (127.12.126.67:42353)
I20250109 21:04:15.817180 23612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:43215
I20250109 21:04:15.822033 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 21:04:15.829953 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:15.823792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.12.126.126:42055)
0109 21:04:15.823940 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:15.823944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:15.824038 (+ 94us) server_negotiation.cc:407] Connection header received
0109 21:04:15.824139 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:15.824142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:15.824182 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:15.824245 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:04:15.824946 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.825424 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:15.826029 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.826144 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:15.827365 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:04:15.827379 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:04:15.827381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:04:15.827399 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:04:15.828641 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:04:15.828956 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:04:15.828958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:04:15.828960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:04:15.828993 (+ 33us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:04:15.829150 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:04:15.829152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:04:15.829153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:04:15.829300 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:04:15.829395 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:15.829685 (+ 290us) server_negotiation.cc:299] Negotiation successful
0109 21:04:15.829782 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20250109 21:04:15.832854 23612 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:04:15.835409 23612 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:04:15.847296 24082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:15.843214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:33103 (local address 127.0.0.1:41458)
0109 21:04:15.844318 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:15.844330 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:15.844399 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:15.844788 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:15.844794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:15.844813 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:04:15.845094 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:15.845102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.845993 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:15.845996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:15.846753 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:15.846762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.846960 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:15.847015 (+ 55us) client_negotiation.cc:769] Sending connection context
0109 21:04:15.847106 (+ 91us) client_negotiation.cc:240] Negotiation successful
0109 21:04:15.847185 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"thread_start_us":107,"threads_started":1}
I20250109 21:04:15.847860 24085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:15.843089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:36707 (local address 127.0.0.1:57646)
0109 21:04:15.843881 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:15.843921 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:15.844048 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:15.844816 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:15.844819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:15.844836 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:04:15.845095 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:15.845105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.846511 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:15.846514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:15.847494 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:15.847504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.847614 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:15.847635 (+ 21us) client_negotiation.cc:769] Sending connection context
0109 21:04:15.847688 (+ 53us) client_negotiation.cc:240] Negotiation successful
0109 21:04:15.847745 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":90,"threads_started":1}
I20250109 21:04:15.848395 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:15.843990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.12.126.65:33103)
0109 21:04:15.844392 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:15.844397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:15.844418 (+ 21us) server_negotiation.cc:407] Connection header received
0109 21:04:15.844504 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:15.844509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:15.844660 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:15.844778 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:04:15.845221 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.845892 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:15.847618 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.848024 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:15.848142 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:15.848214 (+ 72us) server_negotiation.cc:299] Negotiation successful
0109 21:04:15.848303 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":107,"threads_started":1}
I20250109 21:04:15.848623 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:15.843474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:42353 (local address 127.0.0.1:60158)
0109 21:04:15.844429 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:15.844443 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:15.844533 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:15.844945 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:15.844948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:15.844961 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:04:15.845136 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:15.845145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.846511 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:15.846514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:15.847494 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:15.847504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.848344 (+ 840us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:15.848363 (+ 19us) client_negotiation.cc:769] Sending connection context
0109 21:04:15.848446 (+ 83us) client_negotiation.cc:240] Negotiation successful
0109 21:04:15.848512 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"thread_start_us":87,"threads_started":1}
I20250109 21:04:15.848767 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:15.843862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60158 (local address 127.12.126.67:42353)
0109 21:04:15.844210 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:15.844217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:15.844523 (+ 306us) server_negotiation.cc:407] Connection header received
0109 21:04:15.844671 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:15.844676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:15.844825 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:15.844966 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:04:15.845284 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.846374 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:15.847659 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.848171 (+ 512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:15.848276 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:15.848526 (+ 250us) server_negotiation.cc:299] Negotiation successful
0109 21:04:15.848627 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":88,"threads_started":1}
I20250109 21:04:15.849036 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:15.843232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.12.126.66:36707)
0109 21:04:15.843628 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:15.843634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:15.844199 (+ 565us) server_negotiation.cc:407] Connection header received
0109 21:04:15.844299 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:15.844304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:15.844511 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:15.844661 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:04:15.845208 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.846374 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:15.848099 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:15.848620 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:15.848734 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:15.848802 (+ 68us) server_negotiation.cc:299] Negotiation successful
0109 21:04:15.848890 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":101,"threads_started":1}
I20250109 21:04:15.851253 24005 tablet_service.cc:1467] Processing CreateTablet for tablet 0f0f3338aef3432eb2155d993add8405 (DEFAULT_TABLE table=test-table [id=e6006963e9d4492fa20c5436e6777ff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:04:15.851258 23871 tablet_service.cc:1467] Processing CreateTablet for tablet 0f0f3338aef3432eb2155d993add8405 (DEFAULT_TABLE table=test-table [id=e6006963e9d4492fa20c5436e6777ff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:04:15.851271 23737 tablet_service.cc:1467] Processing CreateTablet for tablet 0f0f3338aef3432eb2155d993add8405 (DEFAULT_TABLE table=test-table [id=e6006963e9d4492fa20c5436e6777ff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:04:15.852108 24005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f0f3338aef3432eb2155d993add8405. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:04:15.852118 23737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f0f3338aef3432eb2155d993add8405. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:04:15.852118 23871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f0f3338aef3432eb2155d993add8405. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:04:15.858434 24088 tablet_bootstrap.cc:492] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3: Bootstrap starting.
I20250109 21:04:15.859007 24089 tablet_bootstrap.cc:492] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54: Bootstrap starting.
I20250109 21:04:15.860455 24088 tablet_bootstrap.cc:654] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3: Neither blocks nor log segments found. Creating new log.
I20250109 21:04:15.860594 24089 tablet_bootstrap.cc:654] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54: Neither blocks nor log segments found. Creating new log.
I20250109 21:04:15.860814 24090 tablet_bootstrap.cc:492] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba: Bootstrap starting.
I20250109 21:04:15.861622 24089 log.cc:826] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54: Log is configured to *not* fsync() on all Append() calls
I20250109 21:04:15.861682 24088 log.cc:826] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3: Log is configured to *not* fsync() on all Append() calls
I20250109 21:04:15.862843 24090 tablet_bootstrap.cc:654] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba: Neither blocks nor log segments found. Creating new log.
I20250109 21:04:15.863590 24088 tablet_bootstrap.cc:492] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3: No bootstrap required, opened a new log
I20250109 21:04:15.863629 24089 tablet_bootstrap.cc:492] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54: No bootstrap required, opened a new log
I20250109 21:04:15.863760 24088 ts_tablet_manager.cc:1397] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:04:15.863801 24089 ts_tablet_manager.cc:1397] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:04:15.864289 24090 log.cc:826] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba: Log is configured to *not* fsync() on all Append() calls
I20250109 21:04:15.866061 24090 tablet_bootstrap.cc:492] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba: No bootstrap required, opened a new log
I20250109 21:04:15.866241 24090 ts_tablet_manager.cc:1397] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20250109 21:04:15.866360 24088 raft_consensus.cc:357] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c196489edf9346c4bd5c29b020a4d5ba" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36707 } } peers { permanent_uuid: "05493057fa3b4f439fa5089a7d4fbf54" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 33103 } } peers { permanent_uuid: "b222cc54fdc449958c62721027cc07c3" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 42353 } }
I20250109 21:04:15.866536 24088 raft_consensus.cc:383] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:04:15.866606 24088 raft_consensus.cc:738] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b222cc54fdc449958c62721027cc07c3, State: Initialized, Role: FOLLOWER
I20250109 21:04:15.866835 24089 raft_consensus.cc:357] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c196489edf9346c4bd5c29b020a4d5ba" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36707 } } peers { permanent_uuid: "05493057fa3b4f439fa5089a7d4fbf54" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 33103 } } peers { permanent_uuid: "b222cc54fdc449958c62721027cc07c3" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 42353 } }
I20250109 21:04:15.867027 24089 raft_consensus.cc:383] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:04:15.867122 24089 raft_consensus.cc:738] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05493057fa3b4f439fa5089a7d4fbf54, State: Initialized, Role: FOLLOWER
I20250109 21:04:15.867055 24088 consensus_queue.cc:260] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c196489edf9346c4bd5c29b020a4d5ba" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36707 } } peers { permanent_uuid: "05493057fa3b4f439fa5089a7d4fbf54" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 33103 } } peers { permanent_uuid: "b222cc54fdc449958c62721027cc07c3" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 42353 } }
I20250109 21:04:15.867527 24089 consensus_queue.cc:260] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c196489edf9346c4bd5c29b020a4d5ba" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36707 } } peers { permanent_uuid: "05493057fa3b4f439fa5089a7d4fbf54" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 33103 } } peers { permanent_uuid: "b222cc54fdc449958c62721027cc07c3" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 42353 } }
I20250109 21:04:15.867832 24088 ts_tablet_manager.cc:1428] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250109 21:04:15.868191 23803 heartbeater.cc:502] Master 127.12.126.126:42055 was elected leader, sending a full tablet report...
I20250109 21:04:15.868189 24071 heartbeater.cc:502] Master 127.12.126.126:42055 was elected leader, sending a full tablet report...
I20250109 21:04:15.868335 24089 ts_tablet_manager.cc:1428] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250109 21:04:15.869359 24090 raft_consensus.cc:357] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c196489edf9346c4bd5c29b020a4d5ba" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36707 } } peers { permanent_uuid: "05493057fa3b4f439fa5089a7d4fbf54" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 33103 } } peers { permanent_uuid: "b222cc54fdc449958c62721027cc07c3" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 42353 } }
I20250109 21:04:15.871116 24090 raft_consensus.cc:383] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:04:15.871174 24090 raft_consensus.cc:738] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c196489edf9346c4bd5c29b020a4d5ba, State: Initialized, Role: FOLLOWER
I20250109 21:04:15.871659 24090 consensus_queue.cc:260] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c196489edf9346c4bd5c29b020a4d5ba" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36707 } } peers { permanent_uuid: "05493057fa3b4f439fa5089a7d4fbf54" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 33103 } } peers { permanent_uuid: "b222cc54fdc449958c62721027cc07c3" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 42353 } }
I20250109 21:04:15.872293 23937 heartbeater.cc:502] Master 127.12.126.126:42055 was elected leader, sending a full tablet report...
I20250109 21:04:15.872509 24090 ts_tablet_manager.cc:1428] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20250109 21:04:15.882884 23804 tablet.cc:2367] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250109 21:04:16.052728 24072 tablet.cc:2367] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250109 21:04:16.082454 23938 tablet.cc:2367] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:04:16.100174 24095 raft_consensus.cc:491] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:04:16.100416 24095 raft_consensus.cc:513] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c196489edf9346c4bd5c29b020a4d5ba" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36707 } } peers { permanent_uuid: "05493057fa3b4f439fa5089a7d4fbf54" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 33103 } } peers { permanent_uuid: "b222cc54fdc449958c62721027cc07c3" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 42353 } }
I20250109 21:04:16.101554 24095 leader_election.cc:290] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c196489edf9346c4bd5c29b020a4d5ba (127.12.126.66:36707), b222cc54fdc449958c62721027cc07c3 (127.12.126.67:42353)
I20250109 21:04:16.104700 24097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:16.101819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:42353 (local address 127.12.126.65:35733)
0109 21:04:16.102199 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:16.102210 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:16.102351 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:16.102590 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:16.102593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:16.102630 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:04:16.102924 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:16.102931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:16.103643 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:16.103646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:16.104286 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:16.104295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:16.104402 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:16.104424 (+ 22us) client_negotiation.cc:769] Sending connection context
0109 21:04:16.104475 (+ 51us) client_negotiation.cc:240] Negotiation successful
0109 21:04:16.104542 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"spinlock_wait_cycles":768,"thread_start_us":132,"threads_started":1}
I20250109 21:04:16.104705 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:16.101802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:36707 (local address 127.12.126.65:56067)
0109 21:04:16.102153 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:16.102165 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:04:16.102320 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:04:16.102561 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:04:16.102564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:04:16.102580 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:04:16.102895 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:16.102903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:16.103596 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:04:16.103601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:04:16.104275 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:04:16.104284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:04:16.104404 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:16.104424 (+ 20us) client_negotiation.cc:769] Sending connection context
0109 21:04:16.104476 (+ 52us) client_negotiation.cc:240] Negotiation successful
0109 21:04:16.104531 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"spinlock_wait_cycles":19968,"thread_start_us":90,"threads_started":1}
I20250109 21:04:16.105141 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:16.101933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:35733 (local address 127.12.126.67:42353)
0109 21:04:16.102074 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:16.102080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:16.102318 (+ 238us) server_negotiation.cc:407] Connection header received
0109 21:04:16.102451 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:16.102455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:16.102498 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:16.102569 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:04:16.103030 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:16.103529 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:16.104398 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:16.104877 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:16.104923 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:16.104978 (+ 55us) server_negotiation.cc:299] Negotiation successful
0109 21:04:16.105040 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20250109 21:04:16.105150 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:16.101894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:56067 (local address 127.12.126.66:36707)
0109 21:04:16.102086 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:16.102094 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:16.102248 (+ 154us) server_negotiation.cc:407] Connection header received
0109 21:04:16.102434 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:16.102437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:16.102479 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:16.102545 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:04:16.103005 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:16.103497 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:16.104403 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:16.104880 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:16.104916 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:16.104972 (+ 56us) server_negotiation.cc:299] Negotiation successful
0109 21:04:16.105039 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20250109 21:04:16.105690 24025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f0f3338aef3432eb2155d993add8405" candidate_uuid: "05493057fa3b4f439fa5089a7d4fbf54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b222cc54fdc449958c62721027cc07c3" is_pre_election: true
I20250109 21:04:16.105692 23891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f0f3338aef3432eb2155d993add8405" candidate_uuid: "05493057fa3b4f439fa5089a7d4fbf54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c196489edf9346c4bd5c29b020a4d5ba" is_pre_election: true
I20250109 21:04:16.105952 23891 raft_consensus.cc:2463] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05493057fa3b4f439fa5089a7d4fbf54 in term 0.
I20250109 21:04:16.105953 24025 raft_consensus.cc:2463] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05493057fa3b4f439fa5089a7d4fbf54 in term 0.
I20250109 21:04:16.106329 23693 leader_election.cc:304] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05493057fa3b4f439fa5089a7d4fbf54, b222cc54fdc449958c62721027cc07c3; no voters:
I20250109 21:04:16.106554 24095 raft_consensus.cc:2798] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:04:16.106626 24095 raft_consensus.cc:491] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:04:16.106675 24095 raft_consensus.cc:3054] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:04:16.107631 24095 raft_consensus.cc:513] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c196489edf9346c4bd5c29b020a4d5ba" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36707 } } peers { permanent_uuid: "05493057fa3b4f439fa5089a7d4fbf54" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 33103 } } peers { permanent_uuid: "b222cc54fdc449958c62721027cc07c3" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 42353 } }
I20250109 21:04:16.107946 24095 leader_election.cc:290] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [CANDIDATE]: Term 1 election: Requested vote from peers c196489edf9346c4bd5c29b020a4d5ba (127.12.126.66:36707), b222cc54fdc449958c62721027cc07c3 (127.12.126.67:42353)
I20250109 21:04:16.108239 23891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f0f3338aef3432eb2155d993add8405" candidate_uuid: "05493057fa3b4f439fa5089a7d4fbf54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c196489edf9346c4bd5c29b020a4d5ba"
I20250109 21:04:16.108286 24025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f0f3338aef3432eb2155d993add8405" candidate_uuid: "05493057fa3b4f439fa5089a7d4fbf54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b222cc54fdc449958c62721027cc07c3"
I20250109 21:04:16.108358 23891 raft_consensus.cc:3054] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:04:16.108397 24025 raft_consensus.cc:3054] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:04:16.109268 23891 raft_consensus.cc:2463] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05493057fa3b4f439fa5089a7d4fbf54 in term 1.
I20250109 21:04:16.109311 24025 raft_consensus.cc:2463] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05493057fa3b4f439fa5089a7d4fbf54 in term 1.
I20250109 21:04:16.109587 23690 leader_election.cc:304] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05493057fa3b4f439fa5089a7d4fbf54, c196489edf9346c4bd5c29b020a4d5ba; no voters:
I20250109 21:04:16.109761 24095 raft_consensus.cc:2798] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:04:16.109979 24095 raft_consensus.cc:695] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [term 1 LEADER]: Becoming Leader. State: Replica: 05493057fa3b4f439fa5089a7d4fbf54, State: Running, Role: LEADER
I20250109 21:04:16.110347 24095 consensus_queue.cc:237] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c196489edf9346c4bd5c29b020a4d5ba" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36707 } } peers { permanent_uuid: "05493057fa3b4f439fa5089a7d4fbf54" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 33103 } } peers { permanent_uuid: "b222cc54fdc449958c62721027cc07c3" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 42353 } }
I20250109 21:04:16.113525 23611 catalog_manager.cc:5526] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05493057fa3b4f439fa5089a7d4fbf54 (127.12.126.65). New cstate: current_term: 1 leader_uuid: "05493057fa3b4f439fa5089a7d4fbf54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c196489edf9346c4bd5c29b020a4d5ba" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05493057fa3b4f439fa5089a7d4fbf54" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 33103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b222cc54fdc449958c62721027cc07c3" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 42353 } health_report { overall_health: UNKNOWN } } }
I20250109 21:04:16.143992 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:16.140930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.12.126.65:33103)
0109 21:04:16.141096 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0109 21:04:16.141101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:04:16.141114 (+ 13us) server_negotiation.cc:407] Connection header received
0109 21:04:16.141195 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:04:16.141197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:04:16.141250 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:04:16.141362 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:04:16.141717 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:16.142183 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:04:16.142862 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:04:16.143053 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:04:16.143150 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:04:16.143618 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:04:16.143728 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0109 21:04:16.143811 (+ 83us) server_negotiation.cc:299] Negotiation successful
0109 21:04:16.143861 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20250109 21:04:16.149410 23891 raft_consensus.cc:1270] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba [term 1 FOLLOWER]: Refusing update from remote peer 05493057fa3b4f439fa5089a7d4fbf54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:04:16.149408 24025 raft_consensus.cc:1270] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3 [term 1 FOLLOWER]: Refusing update from remote peer 05493057fa3b4f439fa5089a7d4fbf54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:04:16.150072 24095 consensus_queue.cc:1035] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b222cc54fdc449958c62721027cc07c3" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 42353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:04:16.150270 24099 consensus_queue.cc:1035] T 0f0f3338aef3432eb2155d993add8405 P 05493057fa3b4f439fa5089a7d4fbf54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c196489edf9346c4bd5c29b020a4d5ba" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:04:16.159845 24104 mvcc.cc:204] Tried to move back new op lower bound from 7112526463581106176 to 7112526463430664192. Current Snapshot: MvccSnapshot[applied={T|T < 7112526463581106176 or (T in {7112526463581106176})}]
I20250109 21:04:16.160056 24103 mvcc.cc:204] Tried to move back new op lower bound from 7112526463581106176 to 7112526463430664192. Current Snapshot: MvccSnapshot[applied={T|T < 7112526463581106176}]
I20250109 21:04:16.165907 23610 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35816:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:04:16.166093 23610 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35816:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:04:16.168458 23610 catalog_manager.cc:5813] T 00000000000000000000000000000000 P d7099a8594a74df9bad55509317089ea: Sending DeleteTablet for 3 replicas of tablet 0f0f3338aef3432eb2155d993add8405
I20250109 21:04:16.169162 23737 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f0f3338aef3432eb2155d993add8405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:04:16 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:41458
I20250109 21:04:16.169164 23871 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f0f3338aef3432eb2155d993add8405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:04:16 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:57646
I20250109 21:04:16.169174 24005 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f0f3338aef3432eb2155d993add8405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:04:16 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:60158
I20250109 21:04:16.169643 24112 tablet_replica.cc:331] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3: stopping tablet replica
I20250109 21:04:16.169904 24112 raft_consensus.cc:2238] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:04:16.170233 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 23674
I20250109 21:04:16.170243 24112 raft_consensus.cc:2267] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:04:16.171314 24114 tablet_replica.cc:331] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba: stopping tablet replica
I20250109 21:04:16.171661 24114 raft_consensus.cc:2238] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:04:16.171970 24114 raft_consensus.cc:2267] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:04:16.173404 24114 ts_tablet_manager.cc:1905] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:04:16.175489 24114 ts_tablet_manager.cc:1918] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:04:16.175567 24114 log.cc:1198] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/0f0f3338aef3432eb2155d993add8405
I20250109 21:04:16.175843 24114 ts_tablet_manager.cc:1939] T 0f0f3338aef3432eb2155d993add8405 P c196489edf9346c4bd5c29b020a4d5ba: Deleting consensus metadata
W20250109 21:04:16.176630 23598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.126.65:33103 (error 108)
I20250109 21:04:16.176996 23596 catalog_manager.cc:4872] TS c196489edf9346c4bd5c29b020a4d5ba (127.12.126.66:36707): tablet 0f0f3338aef3432eb2155d993add8405 (table test-table [id=e6006963e9d4492fa20c5436e6777ff9]) successfully deleted
I20250109 21:04:16.177624 24085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:04:16.177314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:33103 (local address 127.0.0.1:41478)
0109 21:04:16.177456 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0109 21:04:16.177523 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.65:33103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20250109 21:04:16.177681 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 23808
W20250109 21:04:16.177742 23598 catalog_manager.cc:4615] TS 05493057fa3b4f439fa5089a7d4fbf54 (127.12.126.65:33103): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f0f3338aef3432eb2155d993add8405: Network error: Client connection negotiation failed: client connection to 127.12.126.65:33103: connect: Connection refused (error 111)
I20250109 21:04:16.180202 24112 ts_tablet_manager.cc:1905] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:04:16.182176 24112 ts_tablet_manager.cc:1918] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:04:16.182288 24112 log.cc:1198] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/0f0f3338aef3432eb2155d993add8405
I20250109 21:04:16.182726 24112 ts_tablet_manager.cc:1939] T 0f0f3338aef3432eb2155d993add8405 P b222cc54fdc449958c62721027cc07c3: Deleting consensus metadata
I20250109 21:04:16.183648 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 23942
I20250109 21:04:16.184166 23599 catalog_manager.cc:4872] TS b222cc54fdc449958c62721027cc07c3 (127.12.126.67:42353): tablet 0f0f3338aef3432eb2155d993add8405 (table test-table [id=e6006963e9d4492fa20c5436e6777ff9]) successfully deleted
I20250109 21:04:16.188725 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 23580
2025-01-09T21:04:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2352 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:04:16 dist-test-slave-g0zb krb5kdc[24118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:04:16 dist-test-slave-g0zb krb5kdc[24118](info): set up 2 sockets
Jan 09 21:04:16 dist-test-slave-g0zb krb5kdc[24118](info): commencing operation
krb5kdc: starting...
W20250109 21:04:17.200202 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:04:17 dist-test-slave-g0zb krb5kdc[24118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:04:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:04:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.126.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.126.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.126.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1736456573526088-12793-0/krb5kdc/rangeradmin_127.12.126.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.126.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1736456573526088-12793-0/krb5kdc/rangeradmin_127.12.126.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.126.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.126.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.126.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1736456573526088-12793-0/krb5kdc/rangerlookup_127.12.126.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.126.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1736456573526088-12793-0/krb5kdc/rangerlookup_127.12.126.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.126.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.126.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.126.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1736456573526088-12793-0/krb5kdc/HTTP_127.12.126.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.126.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1736456573526088-12793-0/krb5kdc/HTTP_127.12.126.84@KRBTEST.COM.keytab.
I20250109 21:04:17.369783 12793 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1736456573526088-12793-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6EjPKb/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1736456573526088-12793-0/minicluster-data/postgres -l logfile start
2025-01-09 21:04:19.955 UTC [24148] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-01-09 21:04:19.955 UTC [24148] LOG: listening on IPv4 address "127.12.126.84", port 35673
2025-01-09 21:04:19.959 UTC [24148] LOG: listening on Unix socket "/tmp/.s.PGSQL.35673"
2025-01-09 21:04:19.964 UTC [24153] LOG: database system was shut down at 2025-01-09 21:04:18 UTC
2025-01-09 21:04:19.969 UTC [24148] LOG: database system is ready to accept connections
I20250109 21:04:20.900110 12793 mini_postgres.cc:96] Postgres bound to 35673
2025-01-09 21:04:20.906 UTC [24158] FATAL: database "slave" does not exist
127.12.126.84:35673 - accepting connections
I20250109 21:04:20.907253 12793 mini_ranger.cc:161] Starting Ranger...
I20250109 21:04:20.926256 12793 mini_ranger.cc:84] Created miniranger Postgres user
I20250109 21:04:20.997735 12793 mini_ranger.cc:87] Created ranger Postgres database
I20250109 21:04:20.997846 12793 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1736456573526088-12793-0/minicluster-data/ranger-admin
2025-01-09 21:04:21,287 [I] DB FLAVOR :POSTGRES
2025-01-09 21:04:21,288 [I] --------- Verifying Ranger DB connection ---------
2025-01-09 21:04:21,288 [I] Checking connection..
2025-01-09 21:04:21,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-01-09 21:04:21,623 [I] Checking connection passed.
2025-01-09 21:04:21,624 [I] --------- Verifying version history table ---------
2025-01-09 21:04:21,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-01-09 21:04:21,927 [I] Table x_db_version_h does not exist in database ranger
2025-01-09 21:04:21,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-01-09 21:04:22,264 [I] Table x_db_version_h does not exist in database ranger
2025-01-09 21:04:22,264 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-01-09 21:04:22,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-01-09 21:04:22.576 UTC [24225] WARNING: there is no transaction in progress
2025-01-09 21:04:22,583 [I] create_dbversion_catalog.sql file imported successfully
2025-01-09 21:04:22,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-01-09 21:04:22,912 [I] Table x_db_version_h already exists in database 'ranger'
2025-01-09 21:04:22,912 [I] --------- Importing Ranger Core DB Schema ---------
2025-01-09 21:04:22,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-01-09 21:04:23,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-01-09 21:04:23,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g0zb.c.gcp-upstream.internal','N') ;"
2025-01-09 21:04:23,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-01-09 21:04:24,167 [I] Table x_portal_user does not exist in database ranger
2025-01-09 21:04:24,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-01-09 21:04:24,477 [I] Table x_policy_ref_group does not exist in database ranger
2025-01-09 21:04:24,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-01-09 21:04:24,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-01-09 21:04:25,081 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-01-09 21:04:25,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-01-09 21:04:25.890 UTC [24342] WARNING: there is no transaction in progress
2025-01-09 21:04:25.901 UTC [24342] WARNING: there is no transaction in progress
2025-01-09 21:04:25.913 UTC [24342] WARNING: there is no transaction in progress
2025-01-09 21:04:25.923 UTC [24342] WARNING: there is no transaction in progress
2025-01-09 21:04:25.935 UTC [24342] WARNING: there is no transaction in progress
2025-01-09 21:04:25.945 UTC [24342] WARNING: there is no transaction in progress
2025-01-09 21:04:25.991 UTC [24342] WARNING: there is no transaction in progress
2025-01-09 21:04:25.996 UTC [24342] WARNING: there is no transaction in progress
2025-01-09 21:04:26.006 UTC [24342] WARNING: there is no transaction in progress
2025-01-09 21:04:26.015 UTC [24342] WARNING: there is no transaction in progress
2025-01-09 21:04:26.024 UTC [24342] WARNING: there is no transaction in progress
2025-01-09 21:04:26.033 UTC [24342] WARNING: there is no transaction in progress
2025-01-09 21:04:26.041 UTC [24342] WARNING: there is no transaction in progress
2025-01-09 21:04:26.045 UTC [24342] WARNING: there is no transaction in progress
2025-01-09 21:04:26.049 UTC [24342] WARNING: there is no transaction in progress
2025-01-09 21:04:26.052 UTC [24342] WARNING: there is no transaction in progress
2025-01-09 21:04:26,444 [I] ranger_core_db_postgres.sql file imported successfully
2025-01-09 21:04:26,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-01-09 21:04:26,756 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-01-09 21:04:26,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-01-09 21:04:27,065 [I] Table x_portal_user already exists in database 'ranger'
2025-01-09 21:04:27,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-01-09 21:04:27,383 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-01-09 21:04:27,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-01-09 21:04:27,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-01-09 21:04:28,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g0zb.c.gcp-upstream.internal';"
2025-01-09 21:04:28,311 [I] CORE_DB_SCHEMA import status has been updated
2025-01-09 21:04:28,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:35673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-01-09 21:04:28,621 [I] DB_PATCHES have already been applied
I20250109 21:04:28.627051 12793 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1736456573526088-12793-0/minicluster-data/ranger-admin:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6EjPKb/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20250109 21:04:28.627121 12793 mini_ranger.cc:193] Using host: 127.12.126.84
I20250109 21:04:28.630132 12793 mini_ranger.cc:227] Ranger admin URL: http://127.12.126.84:44967
Jan 09, 2025 9:04:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jan 09, 2025 9:04:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jan 09, 2025 9:04:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1736456573526088-12793-0/minicluster-data/ranger-admin/ews/webapp
Jan 09, 2025 9:04:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1736456573526088-12793-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jan 09, 2025 9:04:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1736456573526088-12793-0/minicluster-data/ranger-admin/ews/webapp] .....
Jan 09, 2025 9:04:29 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Jan 09, 2025 9:04:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1736456573526088-12793-0/minicluster-data/ranger-admin/ews/webapp].
Jan 09, 2025 9:04:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Jan 09, 2025 9:04:29 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-44967"]
Jan 09, 2025 9:04:29 PM org.apache.coyote.AbstractProtocol init
SEVERE: Failed to initialize end point associated with ProtocolHandler ["http-bio-44967"]
java.net.BindException: Address already in use (Bind failed) <null>:44967
at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:414)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:750)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:467)
at org.apache.coyote.http11.AbstractHttp11JsseProtocol.init(AbstractHttp11JsseProtocol.java:119)
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1030)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:560)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:826)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:173)
at org.apache.catalina.startup.Tomcat.start(Tomcat.java:469)
at org.apache.ranger.server.tomcat.EmbeddedServer.startServer(EmbeddedServer.java:322)
at org.apache.ranger.server.tomcat.EmbeddedServer.start(EmbeddedServer.java:290)
at org.apache.ranger.server.tomcat.EmbeddedServer.main(EmbeddedServer.java:84)
Caused by: java.net.BindException: Address already in use (Bind failed)
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
at java.net.ServerSocket.bind(ServerSocket.java:390)
at java.net.ServerSocket.<init>(ServerSocket.java:252)
at java.net.ServerSocket.<init>(ServerSocket.java:196)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.createSocket(DefaultServerSocketFactory.java:49)
at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:401)
... 14 more
Jan 09, 2025 9:04:29 PM org.apache.catalina.core.StandardService initInternal
SEVERE: Failed to initialize connector [Connector[HTTP/1.1-44967]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1032)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:560)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:826)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:173)
at org.apache.catalina.startup.Tomcat.start(Tomcat.java:469)
at org.apache.ranger.server.tomcat.EmbeddedServer.startServer(EmbeddedServer.java:322)
at org.apache.ranger.server.tomcat.EmbeddedServer.start(EmbeddedServer.java:290)
at org.apache.ranger.server.tomcat.EmbeddedServer.main(EmbeddedServer.java:84)
Caused by: java.net.BindException: Address already in use (Bind failed) <null>:44967
at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:414)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:750)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:467)
at org.apache.coyote.http11.AbstractHttp11JsseProtocol.init(AbstractHttp11JsseProtocol.java:119)
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1030)
... 10 more
Caused by: java.net.BindException: Address already in use (Bind failed)
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
at java.net.ServerSocket.bind(ServerSocket.java:390)
at java.net.ServerSocket.<init>(ServerSocket.java:252)
at java.net.ServerSocket.<init>(ServerSocket.java:196)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.createSocket(DefaultServerSocketFactory.java:49)
at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:401)
... 14 more
Jan 09, 2025 9:04:29 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Jan 09, 2025 9:04:29 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Jan 09, 2025 9:04:29 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Jan 09, 2025 9:04:29 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Jan 09, 2025 9:04:29 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Jan 09, 2025 9:04:30 PM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Jan 09, 2025 9:04:30 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jan 09, 2025 9:04:30 PM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Jan 09, 2025 9:04:30 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
Jan 09, 2025 9:04:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jan 09, 2025 9:04:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.SecurityZoneREST
Jan 09, 2025 9:04:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jan 09, 2025 9:04:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jan 09, 2025 9:04:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jan 09, 2025 9:04:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jan 09, 2025 9:04:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jan 09, 2025 9:04:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jan 09, 2025 9:04:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jan 09, 2025 9:04:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jan 09, 2025 9:04:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jan 09, 2025 9:04:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jan 09, 2025 9:04:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jan 09, 2025 9:04:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jan 09, 2025 9:04:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jan 09, 2025 9:04:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jan 09, 2025 9:04:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jan 09, 2025 9:04:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20250109 21:06:29.410773 12793 mini_ranger.cc:160] Time spent starting Ranger: real 128.504s user 0.038s sys 0.120s
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/security-itest.cc:1052: Failure
Failed
Bad status: Timed out: Failed to start the Ranger service: process with PID 24436 is not yet bound to any port at the specified addresses; last attempt running lsof returned 'Not found: could not find pattern of a bound port in lsof output: p24436
f310
n127.0.0.1:59484->127.12.126.84:35673
f311
n127.0.0.1:59478->127.12.126.84:35673
f312
n127.0.0.1:59462->127.12.126.84:35673
f313
n127.0.0.1:59492->127.12.126.84:35673
f314
n127.0.0.1:59498->127.12.126.84:35673'
I20250109 21:06:29.411708 12793 mini_ranger.cc:66] Stopping Ranger...
I20250109 21:06:29.452883 12793 mini_ranger.cc:68] Stopped Ranger
2025-01-09 21:06:29.453 UTC [24148] LOG: received smart shutdown request
2025-01-09 21:06:29.456 UTC [24148] LOG: background worker "logical replication launcher" (PID 24156) exited with exit code 1
2025-01-09 21:06:29.456 UTC [24151] LOG: shutting down
2025-01-09 21:06:29.458 UTC [24151] LOG: checkpoint starting: shutdown immediate
2025-01-09 21:06:29.874 UTC [24151] LOG: checkpoint complete: wrote 1468 buffers (9.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.023 s, sync=0.388 s, total=0.418 s; sync files=815, longest=0.005 s, average=0.001 s; distance=8396 kB, estimate=8396 kB; lsn=0/1CADC28, redo lsn=0/1CADC28
2025-01-09 21:06:29.880 UTC [24148] LOG: database system is shut down
2025-01-09T21:06:29Z chronyd exiting
I20250109 21:06:29.899729 12793 test_util.cc:181] -----------------------------------------------
I20250109 21:06:29.899806 12793 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1736456573526088-12793-0
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration (133689 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:06:29 dist-test-slave-g0zb krb5kdc[24559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:06:29 dist-test-slave-g0zb krb5kdc[24559](info): set up 2 sockets
Jan 09 21:06:29 dist-test-slave-g0zb krb5kdc[24559](info): commencing operation
krb5kdc: starting...
W20250109 21:06:30.890784 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:06:30 dist-test-slave-g0zb krb5kdc[24559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:06:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:06:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.126.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.126.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.126.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/rangeradmin_127.12.126.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.126.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/rangeradmin_127.12.126.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.126.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.126.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.126.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/rangerlookup_127.12.126.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.126.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/rangerlookup_127.12.126.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.126.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.126.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.126.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/HTTP_127.12.126.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.126.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/HTTP_127.12.126.84@KRBTEST.COM.keytab.
I20250109 21:06:31.047253 12793 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6EjPKb/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/postgres -l logfile start
2025-01-09 21:06:33.519 UTC [24589] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-01-09 21:06:33.519 UTC [24589] LOG: listening on IPv4 address "127.12.126.84", port 38733
2025-01-09 21:06:33.523 UTC [24589] LOG: listening on Unix socket "/tmp/.s.PGSQL.38733"
2025-01-09 21:06:33.530 UTC [24594] LOG: database system was shut down at 2025-01-09 21:06:32 UTC
2025-01-09 21:06:33.534 UTC [24589] LOG: database system is ready to accept connections
I20250109 21:06:34.454969 12793 mini_postgres.cc:96] Postgres bound to 38733
2025-01-09 21:06:34.460 UTC [24599] FATAL: database "slave" does not exist
127.12.126.84:38733 - accepting connections
I20250109 21:06:34.461376 12793 mini_ranger.cc:161] Starting Ranger...
I20250109 21:06:34.478284 12793 mini_ranger.cc:84] Created miniranger Postgres user
I20250109 21:06:34.547611 12793 mini_ranger.cc:87] Created ranger Postgres database
I20250109 21:06:34.547785 12793 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-admin
2025-01-09 21:06:34,800 [I] DB FLAVOR :POSTGRES
2025-01-09 21:06:34,801 [I] --------- Verifying Ranger DB connection ---------
2025-01-09 21:06:34,801 [I] Checking connection..
2025-01-09 21:06:34,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-01-09 21:06:35,109 [I] Checking connection passed.
2025-01-09 21:06:35,109 [I] --------- Verifying version history table ---------
2025-01-09 21:06:35,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-01-09 21:06:35,414 [I] Table x_db_version_h does not exist in database ranger
2025-01-09 21:06:35,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-01-09 21:06:35,734 [I] Table x_db_version_h does not exist in database ranger
2025-01-09 21:06:35,734 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-01-09 21:06:35,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-01-09 21:06:36.034 UTC [24666] WARNING: there is no transaction in progress
2025-01-09 21:06:36,040 [I] create_dbversion_catalog.sql file imported successfully
2025-01-09 21:06:36,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-01-09 21:06:36,366 [I] Table x_db_version_h already exists in database 'ranger'
2025-01-09 21:06:36,366 [I] --------- Importing Ranger Core DB Schema ---------
2025-01-09 21:06:36,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-01-09 21:06:36,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-01-09 21:06:36,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g0zb.c.gcp-upstream.internal','N') ;"
2025-01-09 21:06:37,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-01-09 21:06:37,610 [I] Table x_portal_user does not exist in database ranger
2025-01-09 21:06:37,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-01-09 21:06:37,929 [I] Table x_policy_ref_group does not exist in database ranger
2025-01-09 21:06:37,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-01-09 21:06:38,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-01-09 21:06:38,585 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-01-09 21:06:38,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-01-09 21:06:39.384 UTC [24783] WARNING: there is no transaction in progress
2025-01-09 21:06:39.399 UTC [24783] WARNING: there is no transaction in progress
2025-01-09 21:06:39.412 UTC [24783] WARNING: there is no transaction in progress
2025-01-09 21:06:39.422 UTC [24783] WARNING: there is no transaction in progress
2025-01-09 21:06:39.433 UTC [24783] WARNING: there is no transaction in progress
2025-01-09 21:06:39.445 UTC [24783] WARNING: there is no transaction in progress
2025-01-09 21:06:39.489 UTC [24783] WARNING: there is no transaction in progress
2025-01-09 21:06:39.495 UTC [24783] WARNING: there is no transaction in progress
2025-01-09 21:06:39.504 UTC [24783] WARNING: there is no transaction in progress
2025-01-09 21:06:39.516 UTC [24783] WARNING: there is no transaction in progress
2025-01-09 21:06:39.527 UTC [24783] WARNING: there is no transaction in progress
2025-01-09 21:06:39.536 UTC [24783] WARNING: there is no transaction in progress
2025-01-09 21:06:39.544 UTC [24783] WARNING: there is no transaction in progress
2025-01-09 21:06:39.548 UTC [24783] WARNING: there is no transaction in progress
2025-01-09 21:06:39.551 UTC [24783] WARNING: there is no transaction in progress
2025-01-09 21:06:39.554 UTC [24783] WARNING: there is no transaction in progress
2025-01-09 21:06:39,935 [I] ranger_core_db_postgres.sql file imported successfully
2025-01-09 21:06:39,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-01-09 21:06:40,253 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-01-09 21:06:40,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-01-09 21:06:40,589 [I] Table x_portal_user already exists in database 'ranger'
2025-01-09 21:06:40,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-01-09 21:06:40,912 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-01-09 21:06:40,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-01-09 21:06:41,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-01-09 21:06:41,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g0zb.c.gcp-upstream.internal';"
2025-01-09 21:06:41,837 [I] CORE_DB_SCHEMA import status has been updated
2025-01-09 21:06:41,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-01-09 21:06:42,152 [I] DB_PATCHES have already been applied
I20250109 21:06:42.158109 12793 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-admin:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6EjPKb/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20250109 21:06:42.158172 12793 mini_ranger.cc:193] Using host: 127.12.126.84
I20250109 21:06:42.161197 12793 mini_ranger.cc:227] Ranger admin URL: http://127.12.126.84:43485
Jan 09, 2025 9:06:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jan 09, 2025 9:06:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jan 09, 2025 9:06:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-admin/ews/webapp
Jan 09, 2025 9:06:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jan 09, 2025 9:06:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-admin/ews/webapp] .....
Jan 09, 2025 9:06:42 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Jan 09, 2025 9:06:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-admin/ews/webapp].
Jan 09, 2025 9:06:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Jan 09, 2025 9:06:43 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-43485"]
Jan 09, 2025 9:06:43 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Jan 09, 2025 9:06:43 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Jan 09, 2025 9:06:43 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
I20250109 21:06:43.270149 12793 mini_ranger.cc:160] Time spent starting Ranger: real 8.809s user 0.000s sys 0.010s
Jan 09, 2025 9:06:43 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Jan 09, 2025 9:06:43 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Jan 09, 2025 9:06:43 PM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Jan 09, 2025 9:06:43 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jan 09, 2025 9:06:43 PM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Jan 09, 2025 9:06:43 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
Jan 09, 2025 9:07:01 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jan 09, 2025 9:07:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.TagREST
Jan 09, 2025 9:07:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jan 09, 2025 9:07:01 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jan 09, 2025 9:07:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jan 09, 2025 9:07:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jan 09, 2025 9:07:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jan 09, 2025 9:07:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jan 09, 2025 9:07:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jan 09, 2025 9:07:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jan 09, 2025 9:07:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jan 09, 2025 9:07:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jan 09, 2025 9:07:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jan 09, 2025 9:07:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jan 09, 2025 9:07:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jan 09, 2025 9:07:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jan 09, 2025 9:07:01 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jan 09, 2025 9:07:02 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Jan 09, 2025 9:07:02 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-43485"]
I20250109 21:07:03.174484 12793 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.126.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.126.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.126.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/rangerkms_127.12.126.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.126.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/rangerkms_127.12.126.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.126.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.126.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.126.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/HTTP_127.12.126.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.126.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/HTTP_127.12.126.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jan 09 21:07:03 dist-test-slave-g0zb krb5kdc[24559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456823, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20250109 21:07:03.249344 12793 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20250109 21:07:03.258308 12793 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20250109 21:07:03.361546 12793 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20250109 21:07:03.361653 12793 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-kms
I20250109 21:07:03.361697 12793 mini_ranger_kms.cc:227] Using postgres at 127.12.126.84:38733
2025-01-09 21:07:03,533 [I] DB FLAVOR :POSTGRES
2025-01-09 21:07:03,533 [I] --------- Verifying Ranger DB connection ---------
2025-01-09 21:07:03,533 [I] Checking connection
2025-01-09 21:07:03,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-01-09 21:07:03,825 [I] connection success
2025-01-09 21:07:03,825 [I] --------- Verifying Ranger DB tables ---------
2025-01-09 21:07:03,825 [I] Verifying table ranger_masterkey in database rangerkms
2025-01-09 21:07:03,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-01-09 21:07:04,129 [I] Table ranger_masterkey does not exist in database rangerkms
2025-01-09 21:07:04,129 [I] --------- Importing Ranger Core DB Schema ---------
2025-01-09 21:07:04,129 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-01-09 21:07:04,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.126.84:38733/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-01-09 21:07:04,447 [I] kms_core_db_postgres.sql DB schema imported successfully
I20250109 21:07:04.875174 12793 mini_ranger_kms.cc:310] Created kms service
I20250109 21:07:05.096971 12793 mini_ranger_kms.cc:326] Created kudu user
I20250109 21:07:05.140885 12793 mini_ranger_kms.cc:343] Created rangerkms user
Jan 09 21:07:05 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456823, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.126.84@KRBTEST.COM
I20250109 21:07:05.513118 12793 mini_ranger_kms.cc:384] Added ranger policy
I20250109 21:07:05.513289 12793 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-kms:/tmp/dist-test-task6EjPKb/build/debug/bin/postgresql.jar:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6EjPKb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20250109 21:07:05.513314 12793 mini_ranger_kms.cc:240] Using host: 127.12.126.84
I20250109 21:07:05.517639 12793 mini_ranger_kms.cc:276] Ranger KMS PID: 24974
I20250109 21:07:05.517743 12793 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.126.84:53167
Jan 09, 2025 9:07:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jan 09, 2025 9:07:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jan 09, 2025 9:07:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jan 09, 2025 9:07:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-kms/ews/webapp] .....
Jan 09, 2025 9:07:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-kms/ews/webapp].
Jan 09, 2025 9:07:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jan 09, 2025 9:07:06 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-53167"]
Jan 09, 2025 9:07:06 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Jan 09, 2025 9:07:06 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Jan 09, 2025 9:07:07 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Jan 09, 2025 9:07:07 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jan 09 21:07:07 dist-test-slave-g0zb krb5kdc[24559](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1736456827, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.126.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:07.763923 12793 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.515s user 0.007s sys 0.149s
I20250109 21:07:07.764081 12793 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20250109 21:07:07.764199 12793 mini_ranger_kms.cc:401] 127.12.126.84:53167/kms/v1/keys
Jan 09 21:07:07 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456827, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.126.84@KRBTEST.COM for HTTP/127.12.126.84@KRBTEST.COM
Jan 09 21:07:08 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456827, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.126.84@KRBTEST.COM for HTTP/127.12.126.84@KRBTEST.COM
Jan 09 21:07:10 dist-test-slave-g0zb krb5kdc[24559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:07:10.367035 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:38813
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:38813
--ranger_config_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.126.84:53167/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:07:10.475351 25025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:10.475561 25025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:10.475610 25025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:10.478529 25025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:07:10.478576 25025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:10.478600 25025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:10.478617 25025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:07:10.478634 25025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:07:10.482928 25025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.126.84:53167/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:38813
--ranger_config_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:38813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:10.483906 25025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:10.484710 25025 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:10.491992 25033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:10.492473 25031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:10.492678 25030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:10.493814 25025 server_base.cc:1034] running on GCE node
I20250109 21:07:10.494221 25025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:10.495071 25025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:10.496212 25025 hybrid_clock.cc:648] HybridClock initialized: now 1736456830496189 us; error 37 us; skew 500 ppm
Jan 09 21:07:10 dist-test-slave-g0zb krb5kdc[24559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456830, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:10.503605 25025 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:07:10.504482 25025 webserver.cc:458] Webserver started at http://127.12.126.126:33059/ using document root <none> and password file <none>
I20250109 21:07:10.504987 25025 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:10.505040 25025 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:10.505209 25025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jan 09 21:07:10 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456830, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for HTTP/127.12.126.84@KRBTEST.COM
I20250109 21:07:10.581233 25025 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "3507bd54644843a5991ce63e234d1925"
format_stamp: "Formatted at 2025-01-09 21:07:10 on dist-test-slave-g0zb"
server_key: "00d55dfa5f288d23e5f7aec988eeb672"
server_key_iv: "0877d243aa3853ecf9fcab4db18c8e2b"
server_key_version: "kuduclusterkey@0"
I20250109 21:07:10.581888 25025 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "3507bd54644843a5991ce63e234d1925"
format_stamp: "Formatted at 2025-01-09 21:07:10 on dist-test-slave-g0zb"
server_key: "00d55dfa5f288d23e5f7aec988eeb672"
server_key_iv: "0877d243aa3853ecf9fcab4db18c8e2b"
server_key_version: "kuduclusterkey@0"
I20250109 21:07:10.585558 25025 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20250109 21:07:10.601110 25040 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:10.602226 25025 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:07:10.602329 25025 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "3507bd54644843a5991ce63e234d1925"
format_stamp: "Formatted at 2025-01-09 21:07:10 on dist-test-slave-g0zb"
server_key: "00d55dfa5f288d23e5f7aec988eeb672"
server_key_iv: "0877d243aa3853ecf9fcab4db18c8e2b"
server_key_version: "kuduclusterkey@0"
I20250109 21:07:10.602439 25025 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:10.621086 25025 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:10.624312 25025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:10.624464 25025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:10.630965 25025 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:38813
I20250109 21:07:10.630973 25092 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:38813 every 8 connection(s)
I20250109 21:07:10.631975 25025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:07:10.634393 25093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:10.634528 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 25025
I20250109 21:07:10.634650 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:07:10.634929 12793 external_mini_cluster.cc:1453] Setting key 2aff77d07502a709cfdd84e3a2c49c58
I20250109 21:07:10.640169 25093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925: Bootstrap starting.
Jan 09 21:07:10 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:07:10.642632 25093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925: Neither blocks nor log segments found. Creating new log.
I20250109 21:07:10.643692 25093 log.cc:826] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:10.645504 25093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925: No bootstrap required, opened a new log
I20250109 21:07:10.646963 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:10.636031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.12.126.126:38813)
0109 21:07:10.636379 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:10.636387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:10.636420 (+ 33us) server_negotiation.cc:407] Connection header received
0109 21:07:10.636990 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:10.637005 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:10.637266 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:10.637560 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:10.638593 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:10.639381 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:10.640219 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:10.640429 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:10.642963 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:10.642979 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:10.642990 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:10.643018 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:10.644806 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:10.645200 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:10.645203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:10.645207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:10.645257 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:10.645459 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:10.645460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:10.645461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:10.645788 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:10.645930 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:10.646183 (+ 253us) server_negotiation.cc:299] Negotiation successful
0109 21:07:10.646369 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":97,"threads_started":1}
I20250109 21:07:10.648211 25093 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3507bd54644843a5991ce63e234d1925" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38813 } }
I20250109 21:07:10.648407 25093 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:10.648463 25093 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3507bd54644843a5991ce63e234d1925, State: Initialized, Role: FOLLOWER
I20250109 21:07:10.648849 25093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3507bd54644843a5991ce63e234d1925" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38813 } }
I20250109 21:07:10.648959 25093 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:07:10.649010 25093 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:07:10.649088 25093 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:10.649746 25093 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3507bd54644843a5991ce63e234d1925" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38813 } }
I20250109 21:07:10.650023 25093 leader_election.cc:304] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3507bd54644843a5991ce63e234d1925; no voters:
I20250109 21:07:10.650236 25093 leader_election.cc:290] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:07:10.650343 25098 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:07:10.650527 25098 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [term 1 LEADER]: Becoming Leader. State: Replica: 3507bd54644843a5991ce63e234d1925, State: Running, Role: LEADER
I20250109 21:07:10.650864 25098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3507bd54644843a5991ce63e234d1925" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38813 } }
I20250109 21:07:10.651234 25093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:07:10.652308 25100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3507bd54644843a5991ce63e234d1925. Latest consensus state: current_term: 1 leader_uuid: "3507bd54644843a5991ce63e234d1925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3507bd54644843a5991ce63e234d1925" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38813 } } }
I20250109 21:07:10.652426 25100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [sys.catalog]: This master's current role is: LEADER
I20250109 21:07:10.652664 25099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3507bd54644843a5991ce63e234d1925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3507bd54644843a5991ce63e234d1925" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38813 } } }
I20250109 21:07:10.652738 25099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [sys.catalog]: This master's current role is: LEADER
I20250109 21:07:10.652786 25107 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:07:10.655602 25107 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:07:10.656024 25093 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20250109 21:07:10.665410 25107 catalog_manager.cc:1348] Generated new cluster ID: 5f12de833dd64a80bc8dbe843ccc6b85
I20250109 21:07:10.665540 25107 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:07:10.694098 25107 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:07:10.695360 25107 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:07:10.710459 25107 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925: Generated new TSK 0
I20250109 21:07:10.711325 25107 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
I20250109 21:07:11.406224 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.126.84:53167/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:38813
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:07:11.541749 25138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:11.541934 25138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:11.542016 25138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:11.545018 25138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:11.545069 25138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:11.545154 25138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:07:11.549721 25138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.126.84:53167/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:38813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:11.550756 25138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:11.551680 25138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:11.559718 25145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:11.560370 25148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:11.561233 25146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:11.562143 25138 server_base.cc:1034] running on GCE node
I20250109 21:07:11.562590 25138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:11.563395 25138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:11.564587 25138 hybrid_clock.cc:648] HybridClock initialized: now 1736456831564561 us; error 33 us; skew 500 ppm
Jan 09 21:07:11 dist-test-slave-g0zb krb5kdc[24559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456831, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:11.568403 25138 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:07:11.569454 25138 webserver.cc:458] Webserver started at http://127.12.126.65:46713/ using document root <none> and password file <none>
I20250109 21:07:11.570137 25138 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:11.570212 25138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:11.570444 25138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jan 09 21:07:11 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456831, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for HTTP/127.12.126.84@KRBTEST.COM
I20250109 21:07:11.584375 25138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "056b12014b8b441187409a9fc324fdbb"
format_stamp: "Formatted at 2025-01-09 21:07:11 on dist-test-slave-g0zb"
server_key: "736c361703688ddc434cd06d57b90888"
server_key_iv: "a08d41552df5e0891b362c82eeb22532"
server_key_version: "kuduclusterkey@0"
I20250109 21:07:11.584892 25138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "056b12014b8b441187409a9fc324fdbb"
format_stamp: "Formatted at 2025-01-09 21:07:11 on dist-test-slave-g0zb"
server_key: "736c361703688ddc434cd06d57b90888"
server_key_iv: "a08d41552df5e0891b362c82eeb22532"
server_key_version: "kuduclusterkey@0"
I20250109 21:07:11.592020 25138 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250109 21:07:11.616272 25155 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:11.617710 25138 fs_manager.cc:730] Time spent opening block manager: real 0.014s user 0.001s sys 0.000s
I20250109 21:07:11.617833 25138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "056b12014b8b441187409a9fc324fdbb"
format_stamp: "Formatted at 2025-01-09 21:07:11 on dist-test-slave-g0zb"
server_key: "736c361703688ddc434cd06d57b90888"
server_key_iv: "a08d41552df5e0891b362c82eeb22532"
server_key_version: "kuduclusterkey@0"
I20250109 21:07:11.617944 25138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:11.639676 25138 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:11.642927 25138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:11.643097 25138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:11.643606 25138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:07:11.644464 25138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:07:11.644517 25138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:11.644572 25138 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:07:11.644604 25138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:11.656499 25138 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:35611
I20250109 21:07:11.657764 25138 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:07:11.658986 25270 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:35611 every 8 connection(s)
I20250109 21:07:11.664242 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 25138
I20250109 21:07:11.664388 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:07:11.664645 12793 external_mini_cluster.cc:1453] Setting key 59461c3d2942a7f66966fa477d9322a2
WARNING: no policy specified for kudu/127.12.126.66@KRBTEST.COM; defaulting to no policy
Jan 09 21:07:11 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456831, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.66@KRBTEST.COM" created.
I20250109 21:07:11.683118 25273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:11.662998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:43991 (local address 127.12.126.126:38813)
0109 21:07:11.664643 (+ 1645us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:11.664648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:11.666119 (+ 1471us) server_negotiation.cc:407] Connection header received
0109 21:07:11.666163 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:11.666166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:11.666227 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:11.666320 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:11.667906 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:11.668753 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:11.669658 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:11.669856 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:11.678530 (+ 8674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:11.678552 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:11.678556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:11.678584 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:11.680714 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:11.681330 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:11.681334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:11.681336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:11.681408 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:11.681768 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:11.681771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:11.681773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:11.681984 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:11.682081 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:11.682763 (+ 682us) server_negotiation.cc:299] Negotiation successful
0109 21:07:11.682930 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1540,"thread_start_us":110,"threads_started":1}
I20250109 21:07:11.684495 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:11.663466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:38813 (local address 127.12.126.65:43991)
0109 21:07:11.664887 (+ 1421us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:11.664931 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:11.665812 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:11.666603 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:11.666614 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:11.666983 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:11.667745 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:11.667762 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:11.668900 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:11.668904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:11.669512 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:11.669523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:11.670134 (+ 611us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:11.676611 (+ 6477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:11.676655 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:11.678310 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:11.680857 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:11.680863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:11.680876 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:11.681199 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:11.681521 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:11.681524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:11.681526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:11.681647 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:11.682137 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:11.682144 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:11.682467 (+ 323us) client_negotiation.cc:769] Sending connection context
0109 21:07:11.683360 (+ 893us) client_negotiation.cc:240] Negotiation successful
0109 21:07:11.683648 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1218,"thread_start_us":107,"threads_started":1}
I20250109 21:07:11.685725 25271 heartbeater.cc:346] Connected to a master server at 127.12.126.126:38813
I20250109 21:07:11.685949 25271 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:11.686532 25271 heartbeater.cc:510] Master 127.12.126.126:38813 requested a full tablet report, sending...
I20250109 21:07:11.688081 25057 ts_manager.cc:194] Registered new tserver with Master: 056b12014b8b441187409a9fc324fdbb (127.12.126.65:35611)
I20250109 21:07:11.689636 25057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:43991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
I20250109 21:07:11.701897 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.126.84:53167/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:38813
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 09 21:07:11 dist-test-slave-g0zb krb5kdc[24559](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1736456831, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20250109 21:07:11.830092 25277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:11.830361 25277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:11.830437 25277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:11.834645 25277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:11.834738 25277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:11.834868 25277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:07:11.840204 25277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.126.84:53167/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:38813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:11.841847 25277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:11.842927 25277 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:11.852609 25283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:11.853613 25284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:11.854071 25286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:11.855022 25277 server_base.cc:1034] running on GCE node
I20250109 21:07:11.855461 25277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:11.856130 25277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:11.857311 25277 hybrid_clock.cc:648] HybridClock initialized: now 1736456831857290 us; error 35 us; skew 500 ppm
Jan 09 21:07:11 dist-test-slave-g0zb krb5kdc[24559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456831, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:11.860832 25277 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:07:11.861897 25277 webserver.cc:458] Webserver started at http://127.12.126.66:38583/ using document root <none> and password file <none>
I20250109 21:07:11.862627 25277 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:11.862747 25277 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:11.863039 25277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jan 09 21:07:11 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456831, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for HTTP/127.12.126.84@KRBTEST.COM
I20250109 21:07:11.876603 25277 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "88acd837bcfa4de98dfd093ce8488660"
format_stamp: "Formatted at 2025-01-09 21:07:11 on dist-test-slave-g0zb"
server_key: "ff2ce11c3b4c4162185756fdc48b1e99"
server_key_iv: "f5fd83c7aeb8d0b1785e0e686aa1e590"
server_key_version: "kuduclusterkey@0"
I20250109 21:07:11.877197 25277 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "88acd837bcfa4de98dfd093ce8488660"
format_stamp: "Formatted at 2025-01-09 21:07:11 on dist-test-slave-g0zb"
server_key: "ff2ce11c3b4c4162185756fdc48b1e99"
server_key_iv: "f5fd83c7aeb8d0b1785e0e686aa1e590"
server_key_version: "kuduclusterkey@0"
I20250109 21:07:11.884055 25277 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250109 21:07:11.902864 25293 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:11.904417 25277 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20250109 21:07:11.904556 25277 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "88acd837bcfa4de98dfd093ce8488660"
format_stamp: "Formatted at 2025-01-09 21:07:11 on dist-test-slave-g0zb"
server_key: "ff2ce11c3b4c4162185756fdc48b1e99"
server_key_iv: "f5fd83c7aeb8d0b1785e0e686aa1e590"
server_key_version: "kuduclusterkey@0"
I20250109 21:07:11.904685 25277 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:11.941345 25277 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:11.945056 25277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:11.945237 25277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:11.945927 25277 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:07:11.946935 25277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:07:11.946997 25277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:11.947050 25277 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:07:11.947080 25277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:11.963733 25277 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:42541
I20250109 21:07:11.966141 25277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:07:11.973079 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 25277
I20250109 21:07:11.973219 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:07:11.973469 12793 external_mini_cluster.cc:1453] Setting key d506cb3611666b48327d7cd7eea134b3
I20250109 21:07:11.972662 25406 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:42541 every 8 connection(s)
WARNING: no policy specified for kudu/127.12.126.67@KRBTEST.COM; defaulting to no policy
Jan 09 21:07:11 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456831, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.67@KRBTEST.COM" created.
I20250109 21:07:12.003628 25273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:11.979655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:60157 (local address 127.12.126.126:38813)
0109 21:07:11.979779 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:11.979783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:11.982360 (+ 2577us) server_negotiation.cc:407] Connection header received
0109 21:07:11.984075 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:11.984079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:11.984136 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:11.984189 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:11.991182 (+ 6993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:11.991967 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:11.992929 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:11.993131 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:11.999412 (+ 6281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:11.999436 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:11.999439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:11.999471 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:12.001258 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:12.001873 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:12.001877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:12.001879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:12.001928 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:12.002347 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:12.002351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:12.002354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:12.002570 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:12.002654 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:12.003339 (+ 685us) server_negotiation.cc:299] Negotiation successful
0109 21:07:12.003466 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250109 21:07:12.004300 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:11.978588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:38813 (local address 127.12.126.66:60157)
0109 21:07:11.982215 (+ 3627us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:11.982272 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:11.983766 (+ 1494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:11.984619 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:11.984630 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:11.985637 (+ 1007us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:11.990979 (+ 5342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:11.991003 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:11.992148 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:11.992152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:11.992784 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:11.992796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:11.993403 (+ 607us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:11.994166 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:11.994202 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:11.999184 (+ 4982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:12.001418 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:12.001424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:12.001437 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:12.001758 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:12.002077 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:12.002080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:12.002082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:12.002206 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:12.002725 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:12.002732 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:12.003079 (+ 347us) client_negotiation.cc:769] Sending connection context
0109 21:07:12.003286 (+ 207us) client_negotiation.cc:240] Negotiation successful
0109 21:07:12.003539 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3395,"thread_start_us":167,"threads_started":1}
I20250109 21:07:12.005417 25407 heartbeater.cc:346] Connected to a master server at 127.12.126.126:38813
I20250109 21:07:12.005648 25407 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:12.006281 25407 heartbeater.cc:510] Master 127.12.126.126:38813 requested a full tablet report, sending...
I20250109 21:07:12.007514 25057 ts_manager.cc:194] Registered new tserver with Master: 88acd837bcfa4de98dfd093ce8488660 (127.12.126.66:42541)
I20250109 21:07:12.008251 25057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:60157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
I20250109 21:07:12.032895 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.126.84:53167/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:38813
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:07:12.187834 25412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:12.188199 25412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:12.188323 25412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:12.192606 25412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:12.192766 25412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:12.192914 25412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:07:12.198725 25412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.126.84:53167/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:38813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:12.200208 25412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:12.201270 25412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
Jan 09 21:07:12 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456831, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for HTTP/127.12.126.84@KRBTEST.COM
W20250109 21:07:12.211489 25420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:12.212505 25417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:12.212749 25412 server_base.cc:1034] running on GCE node
W20250109 21:07:12.214929 25418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:12.215381 25412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:12.215983 25412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:12.217320 25412 hybrid_clock.cc:648] HybridClock initialized: now 1736456832217263 us; error 68 us; skew 500 ppm
Jan 09 21:07:12 dist-test-slave-g0zb krb5kdc[24559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456832, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:12.220710 25412 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:07:12.221694 25412 webserver.cc:458] Webserver started at http://127.12.126.67:43811/ using document root <none> and password file <none>
I20250109 21:07:12.222380 25412 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:12.222434 25412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:12.222612 25412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jan 09 21:07:12 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456832, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for HTTP/127.12.126.84@KRBTEST.COM
I20250109 21:07:12.245829 25412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "71bd1528498448dd9984969588aa4d00"
format_stamp: "Formatted at 2025-01-09 21:07:12 on dist-test-slave-g0zb"
server_key: "973c60926a090ff0f36c4a9d085f613a"
server_key_iv: "5e76632bceb9d42b8c6f8baac7c5360a"
server_key_version: "kuduclusterkey@0"
I20250109 21:07:12.247923 25412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "71bd1528498448dd9984969588aa4d00"
format_stamp: "Formatted at 2025-01-09 21:07:12 on dist-test-slave-g0zb"
server_key: "973c60926a090ff0f36c4a9d085f613a"
server_key_iv: "5e76632bceb9d42b8c6f8baac7c5360a"
server_key_version: "kuduclusterkey@0"
I20250109 21:07:12.253057 25412 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250109 21:07:12.273793 25427 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:12.277688 25412 fs_manager.cc:730] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20250109 21:07:12.277913 25412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "71bd1528498448dd9984969588aa4d00"
format_stamp: "Formatted at 2025-01-09 21:07:12 on dist-test-slave-g0zb"
server_key: "973c60926a090ff0f36c4a9d085f613a"
server_key_iv: "5e76632bceb9d42b8c6f8baac7c5360a"
server_key_version: "kuduclusterkey@0"
I20250109 21:07:12.278146 25412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:12.313971 25412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:12.323547 25412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:12.323910 25412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:12.324796 25412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:07:12.325976 25412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:07:12.326092 25412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:12.326190 25412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:07:12.326262 25412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:12.335984 25412 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:33881
I20250109 21:07:12.338637 25540 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:33881 every 8 connection(s)
I20250109 21:07:12.339674 25412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:07:12.345985 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 25412
I20250109 21:07:12.346117 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:07:12.346375 12793 external_mini_cluster.cc:1453] Setting key bd164ab8402325dad94660b722754b10
Jan 09 21:07:12 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456832, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:07:12.361019 25273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:12.344652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:54779 (local address 127.12.126.126:38813)
0109 21:07:12.344828 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:12.344833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:12.351303 (+ 6470us) server_negotiation.cc:407] Connection header received
0109 21:07:12.352410 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:12.352413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:12.352469 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:12.352559 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:12.353900 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:12.354396 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:12.355045 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:12.355215 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:12.357211 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:12.357228 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:12.357230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:12.357259 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:12.358789 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:12.359418 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:12.359422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:12.359423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:12.359485 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:12.360030 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:12.360032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:12.360033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:12.360170 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:12.360270 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:12.360745 (+ 475us) server_negotiation.cc:299] Negotiation successful
0109 21:07:12.360863 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250109 21:07:12.361688 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:12.345076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:38813 (local address 127.12.126.67:54779)
0109 21:07:12.351167 (+ 6091us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:12.351216 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:12.352176 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:12.352765 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:12.352777 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:12.353142 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:12.353763 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:12.353777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:12.354508 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:12.354511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:12.354921 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:12.354929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:12.355167 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:12.355792 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:12.355818 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:12.357062 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:12.358945 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:12.358950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:12.358959 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:12.359288 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:12.359611 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:12.359614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:12.359616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:12.359747 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:12.360272 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:12.360280 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:12.360519 (+ 239us) client_negotiation.cc:769] Sending connection context
0109 21:07:12.360702 (+ 183us) client_negotiation.cc:240] Negotiation successful
0109 21:07:12.360944 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5852,"thread_start_us":138,"threads_started":1}
I20250109 21:07:12.362732 25541 heartbeater.cc:346] Connected to a master server at 127.12.126.126:38813
I20250109 21:07:12.362965 25541 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:12.363453 25541 heartbeater.cc:510] Master 127.12.126.126:38813 requested a full tablet report, sending...
I20250109 21:07:12.364357 25057 ts_manager.cc:194] Registered new tserver with Master: 71bd1528498448dd9984969588aa4d00 (127.12.126.67:33881)
I20250109 21:07:12.364858 25057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:54779
I20250109 21:07:12.371836 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 21:07:12.372447 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 25138
I20250109 21:07:12.380435 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 25277
I20250109 21:07:12.387651 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 25412
I20250109 21:07:12.395941 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 25025
I20250109 21:07:12.402549 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:38813
--webserver_interface=127.12.126.126
--webserver_port=33059
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:38813
--ranger_config_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.126.84:53167/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.126.84:53167/kms with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:07:12.504453 25544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:12.504716 25544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:12.504792 25544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:12.507790 25544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:07:12.507848 25544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:12.507882 25544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:12.507923 25544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:07:12.507956 25544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:07:12.512084 25544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.126.84:53167/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:38813
--ranger_config_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:38813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=33059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:12.513025 25544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:12.513757 25544 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:12.520684 25550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:12.520870 25549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:12.521369 25552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:12.522153 25544 server_base.cc:1034] running on GCE node
I20250109 21:07:12.522575 25544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:12.523439 25544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:12.524585 25544 hybrid_clock.cc:648] HybridClock initialized: now 1736456832524563 us; error 31 us; skew 500 ppm
Jan 09 21:07:12 dist-test-slave-g0zb krb5kdc[24559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456832, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:12.527130 25544 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:07:12.527912 25544 webserver.cc:458] Webserver started at http://127.12.126.126:33059/ using document root <none> and password file <none>
I20250109 21:07:12.528435 25544 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:12.528497 25544 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jan 09 21:07:12 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456832, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for HTTP/127.12.126.84@KRBTEST.COM
I20250109 21:07:12.571971 25544 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20250109 21:07:12.574000 25560 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:12.575062 25544 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:07:12.575171 25544 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "3507bd54644843a5991ce63e234d1925"
format_stamp: "Formatted at 2025-01-09 21:07:10 on dist-test-slave-g0zb"
server_key: "00d55dfa5f288d23e5f7aec988eeb672"
server_key_iv: "0877d243aa3853ecf9fcab4db18c8e2b"
server_key_version: "kuduclusterkey@0"
I20250109 21:07:12.575579 25544 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:12.585903 25544 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:12.588649 25544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:12.588806 25544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:12.595080 25544 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:38813
I20250109 21:07:12.595103 25612 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:38813 every 8 connection(s)
I20250109 21:07:12.596132 25544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:07:12.598197 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 25544
I20250109 21:07:12.598688 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.126.84:53167/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:35611
--local_ip_for_outbound_sockets=127.12.126.65
--tserver_master_addrs=127.12.126.126:38813
--webserver_port=46713
--webserver_interface=127.12.126.65
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.126.84:53167/kms with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:07:12.600346 25613 sys_catalog.cc:263] Verifying existing consensus state
I20250109 21:07:12.601918 25613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925: Bootstrap starting.
I20250109 21:07:12.617183 25613 log.cc:826] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:12.623057 25613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:07:12.623382 25613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925: Bootstrap complete.
I20250109 21:07:12.626875 25613 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3507bd54644843a5991ce63e234d1925" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38813 } }
I20250109 21:07:12.627163 25613 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3507bd54644843a5991ce63e234d1925, State: Initialized, Role: FOLLOWER
I20250109 21:07:12.627600 25613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3507bd54644843a5991ce63e234d1925" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38813 } }
I20250109 21:07:12.627712 25613 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:07:12.627769 25613 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:07:12.627866 25613 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:07:12.629313 25613 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3507bd54644843a5991ce63e234d1925" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38813 } }
I20250109 21:07:12.629601 25613 leader_election.cc:304] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3507bd54644843a5991ce63e234d1925; no voters:
I20250109 21:07:12.629848 25613 leader_election.cc:290] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250109 21:07:12.629983 25617 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [term 2 FOLLOWER]: Leader election won for term 2
I20250109 21:07:12.630215 25617 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [term 2 LEADER]: Becoming Leader. State: Replica: 3507bd54644843a5991ce63e234d1925, State: Running, Role: LEADER
I20250109 21:07:12.630527 25617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3507bd54644843a5991ce63e234d1925" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38813 } }
I20250109 21:07:12.630861 25613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:07:12.631856 25618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3507bd54644843a5991ce63e234d1925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3507bd54644843a5991ce63e234d1925" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38813 } } }
I20250109 21:07:12.631956 25618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [sys.catalog]: This master's current role is: LEADER
I20250109 21:07:12.632277 25619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3507bd54644843a5991ce63e234d1925. Latest consensus state: current_term: 2 leader_uuid: "3507bd54644843a5991ce63e234d1925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3507bd54644843a5991ce63e234d1925" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38813 } } }
I20250109 21:07:12.632421 25619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925 [sys.catalog]: This master's current role is: LEADER
I20250109 21:07:12.632727 25626 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:07:12.633092 25613 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20250109 21:07:12.638216 25626 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:07:12.639128 25626 catalog_manager.cc:1260] Loaded cluster ID: 5f12de833dd64a80bc8dbe843ccc6b85
I20250109 21:07:12.639218 25626 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:07:12.641640 25626 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:07:12.642477 25626 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925: Loaded TSK: 0
I20250109 21:07:12.643747 25626 catalog_manager.cc:1515] Initializing in-progress tserver states...
W20250109 21:07:12.712474 25615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:12.712801 25615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:12.712905 25615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:12.716535 25615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:12.716643 25615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:12.716773 25615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:07:12.721315 25615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.126.84:53167/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:35611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=46713
--tserver_master_addrs=127.12.126.126:38813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:12.722692 25615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:12.723534 25615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:12.731946 25650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:12.732287 25651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:12.732846 25653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:12.736284 25615 server_base.cc:1034] running on GCE node
I20250109 21:07:12.736857 25615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:12.737572 25615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:12.738730 25615 hybrid_clock.cc:648] HybridClock initialized: now 1736456832738719 us; error 26 us; skew 500 ppm
Jan 09 21:07:12 dist-test-slave-g0zb krb5kdc[24559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456832, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:12.741559 25615 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:07:12.742496 25615 webserver.cc:458] Webserver started at http://127.12.126.65:46713/ using document root <none> and password file <none>
I20250109 21:07:12.743088 25615 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:12.743163 25615 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jan 09 21:07:12 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456832, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for HTTP/127.12.126.84@KRBTEST.COM
I20250109 21:07:12.775524 25615 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20250109 21:07:12.777426 25661 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:12.779270 25615 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:07:12.779382 25615 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "056b12014b8b441187409a9fc324fdbb"
format_stamp: "Formatted at 2025-01-09 21:07:11 on dist-test-slave-g0zb"
server_key: "736c361703688ddc434cd06d57b90888"
server_key_iv: "a08d41552df5e0891b362c82eeb22532"
server_key_version: "kuduclusterkey@0"
I20250109 21:07:12.779985 25615 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:12.835134 25615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:12.838886 25615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:12.839141 25615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:12.841192 25615 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:07:12.842260 25615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:07:12.842360 25615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:12.842494 25615 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:07:12.842589 25615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:12.853607 25615 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:35611
I20250109 21:07:12.854718 25615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:07:12.856741 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 25615
I20250109 21:07:12.857257 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.126.84:53167/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:42541
--local_ip_for_outbound_sockets=127.12.126.66
--tserver_master_addrs=127.12.126.126:38813
--webserver_port=38583
--webserver_interface=127.12.126.66
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.126.84:53167/kms with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:07:12.853642 25774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:35611 every 8 connection(s)
Jan 09 21:07:12 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456832, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:07:12.889916 25778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:12.856886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:38813 (local address 127.12.126.65:44659)
0109 21:07:12.867126 (+ 10240us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:12.867178 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:12.868095 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:12.870349 (+ 2254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:12.870361 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:12.870727 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:12.871504 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:12.871523 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:12.880458 (+ 8935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:12.880466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:12.881142 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:12.881158 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:12.881403 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:12.882270 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:12.882310 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:12.884108 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:12.887001 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:12.887006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:12.887016 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:12.887318 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:12.887812 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:12.887815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:12.887817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:12.887929 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:12.888519 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:12.888525 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:12.888765 (+ 240us) client_negotiation.cc:769] Sending connection context
0109 21:07:12.888926 (+ 161us) client_negotiation.cc:240] Negotiation successful
0109 21:07:12.889156 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10002,"thread_start_us":126,"threads_started":1}
I20250109 21:07:12.899096 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:12.856875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:44659 (local address 127.12.126.126:38813)
0109 21:07:12.868681 (+ 11806us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:12.868699 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:12.868746 (+ 47us) server_negotiation.cc:407] Connection header received
0109 21:07:12.869038 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:12.869055 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:12.869431 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:12.869799 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:12.878950 (+ 9151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:12.880305 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:12.881581 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:12.881833 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:12.884318 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:12.884344 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:12.884356 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:12.884395 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:12.886850 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:12.887567 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:12.887572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:12.887577 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:12.887651 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:12.888043 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:12.888046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:12.888049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:12.888397 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:12.897890 (+ 9493us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:12.898068 (+ 178us) server_negotiation.cc:299] Negotiation successful
0109 21:07:12.898348 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11640,"thread_start_us":124,"threads_started":1}
I20250109 21:07:12.901389 25775 heartbeater.cc:346] Connected to a master server at 127.12.126.126:38813
I20250109 21:07:12.901628 25775 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:12.902254 25775 heartbeater.cc:510] Master 127.12.126.126:38813 requested a full tablet report, sending...
W20250109 21:07:12.904420 25775 heartbeater.cc:643] Failed to heartbeat to 127.12.126.126:38813 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20250109 21:07:12.904542 25775 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:12.904750 25775 heartbeater.cc:510] Master 127.12.126.126:38813 requested a full tablet report, sending...
I20250109 21:07:12.905746 25775 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:12.905941 25775 heartbeater.cc:510] Master 127.12.126.126:38813 requested a full tablet report, sending...
W20250109 21:07:12.906816 25775 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250109 21:07:12.994258 25779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:12.994493 25779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:12.994580 25779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:12.998095 25779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:12.998167 25779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:12.998282 25779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:07:13.002605 25779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.126.84:53167/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:42541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=38583
--tserver_master_addrs=127.12.126.126:38813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:13.003705 25779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:13.004539 25779 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:13.013404 25787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:13.014075 25784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:13.014915 25785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:13.015141 25779 server_base.cc:1034] running on GCE node
I20250109 21:07:13.015547 25779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:13.016076 25779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:13.017231 25779 hybrid_clock.cc:648] HybridClock initialized: now 1736456833017215 us; error 29 us; skew 500 ppm
Jan 09 21:07:13 dist-test-slave-g0zb krb5kdc[24559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456833, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:13.019898 25779 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:07:13.020753 25779 webserver.cc:458] Webserver started at http://127.12.126.66:38583/ using document root <none> and password file <none>
I20250109 21:07:13.021261 25779 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:13.021339 25779 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jan 09 21:07:13 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456833, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for HTTP/127.12.126.84@KRBTEST.COM
I20250109 21:07:13.040776 25779 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:07:13.043254 25795 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:13.044531 25779 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:07:13.044672 25779 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "88acd837bcfa4de98dfd093ce8488660"
format_stamp: "Formatted at 2025-01-09 21:07:11 on dist-test-slave-g0zb"
server_key: "ff2ce11c3b4c4162185756fdc48b1e99"
server_key_iv: "f5fd83c7aeb8d0b1785e0e686aa1e590"
server_key_version: "kuduclusterkey@0"
I20250109 21:07:13.045212 25779 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:13.081923 25779 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:13.085786 25779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:13.085973 25779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:13.087114 25779 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:07:13.088086 25779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:07:13.088147 25779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:13.088197 25779 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:07:13.088232 25779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:13.098868 25779 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:42541
I20250109 21:07:13.100245 25779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:07:13.105248 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 25779
I20250109 21:07:13.105788 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.126.84:53167/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:33881
--local_ip_for_outbound_sockets=127.12.126.67
--tserver_master_addrs=127.12.126.126:38813
--webserver_port=43811
--webserver_interface=127.12.126.67
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.126.84:53167/kms with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:07:13.106715 25908 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:42541 every 8 connection(s)
Jan 09 21:07:13 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456833, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:07:13.122282 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:13.108558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:39851 (local address 127.12.126.126:38813)
0109 21:07:13.108730 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:13.108737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:13.109629 (+ 892us) server_negotiation.cc:407] Connection header received
0109 21:07:13.110972 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:13.110978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:13.111050 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:13.111118 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:13.113001 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:13.113879 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:13.114932 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:13.115156 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:13.117806 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:13.117826 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:13.117830 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:13.117856 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:13.119976 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:13.120577 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:13.120580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:13.120582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:13.120630 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:13.120962 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:13.120966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:13.120968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:13.121156 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:13.121228 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:13.121933 (+ 705us) server_negotiation.cc:299] Negotiation successful
0109 21:07:13.122085 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20250109 21:07:13.123855 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:13.108918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:38813 (local address 127.12.126.66:39851)
0109 21:07:13.109486 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:13.109532 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:13.110661 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:13.111589 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:13.111603 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:13.111967 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:13.112800 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:13.112822 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:13.114059 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:13.114066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:13.114743 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:13.114756 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:13.115417 (+ 661us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:13.116150 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:13.116184 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:13.117627 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:13.120107 (+ 2480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:13.120116 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:13.120129 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:13.120460 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:13.120738 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:13.120744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:13.120747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:13.120860 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:13.121322 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:13.121330 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:13.121636 (+ 306us) client_negotiation.cc:769] Sending connection context
0109 21:07:13.122769 (+ 1133us) client_negotiation.cc:240] Negotiation successful
0109 21:07:13.123050 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":145,"threads_started":1}
I20250109 21:07:13.125167 25909 heartbeater.cc:346] Connected to a master server at 127.12.126.126:38813
I20250109 21:07:13.125391 25909 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:13.125989 25909 heartbeater.cc:510] Master 127.12.126.126:38813 requested a full tablet report, sending...
W20250109 21:07:13.127382 25909 heartbeater.cc:643] Failed to heartbeat to 127.12.126.126:38813 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20250109 21:07:13.127579 25909 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:13.127789 25909 heartbeater.cc:510] Master 127.12.126.126:38813 requested a full tablet report, sending...
I20250109 21:07:13.128554 25909 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:13.128737 25909 heartbeater.cc:510] Master 127.12.126.126:38813 requested a full tablet report, sending...
W20250109 21:07:13.129664 25909 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250109 21:07:13.247081 25911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:13.247393 25911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:13.247493 25911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:13.251989 25911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:13.252079 25911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:13.252182 25911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:07:13.258081 25911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.126.84:53167/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:33881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=43811
--tserver_master_addrs=127.12.126.126:38813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:13.259442 25911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:13.260526 25911 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:13.270613 25919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:13.271202 25918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:13.271785 25921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:13.272177 25911 server_base.cc:1034] running on GCE node
I20250109 21:07:13.272590 25911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:13.273119 25911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:13.274267 25911 hybrid_clock.cc:648] HybridClock initialized: now 1736456833274244 us; error 42 us; skew 500 ppm
Jan 09 21:07:13 dist-test-slave-g0zb krb5kdc[24559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456833, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:13.277034 25911 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:07:13.277891 25911 webserver.cc:458] Webserver started at http://127.12.126.67:43811/ using document root <none> and password file <none>
I20250109 21:07:13.278435 25911 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:13.278507 25911 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jan 09 21:07:13 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456833, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for HTTP/127.12.126.84@KRBTEST.COM
I20250109 21:07:13.293727 25911 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20250109 21:07:13.295624 25929 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:13.297472 25911 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250109 21:07:13.297576 25911 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "71bd1528498448dd9984969588aa4d00"
format_stamp: "Formatted at 2025-01-09 21:07:12 on dist-test-slave-g0zb"
server_key: "973c60926a090ff0f36c4a9d085f613a"
server_key_iv: "5e76632bceb9d42b8c6f8baac7c5360a"
server_key_version: "kuduclusterkey@0"
I20250109 21:07:13.298005 25911 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:13.323691 25911 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:13.327281 25911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:13.327471 25911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:13.328608 25911 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:07:13.329553 25911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:07:13.329638 25911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:13.329710 25911 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:07:13.329741 25911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:13.339957 25911 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:33881
I20250109 21:07:13.339995 26042 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:33881 every 8 connection(s)
I20250109 21:07:13.341032 25911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:07:13.343621 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 25911
Jan 09 21:07:13 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456833, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:07:13.358753 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:13.342939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:43119 (local address 127.12.126.126:38813)
0109 21:07:13.343075 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:13.343079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:13.347725 (+ 4646us) server_negotiation.cc:407] Connection header received
0109 21:07:13.349432 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:13.349436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:13.349502 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:13.349577 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:13.350789 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:13.351598 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:13.352261 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:13.352439 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:13.354279 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:13.354294 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:13.354302 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:13.354324 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:13.356058 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:13.356523 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:13.356527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:13.356529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:13.356582 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:13.357098 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:13.357101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:13.357103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:13.357284 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:13.357363 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:13.358460 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0109 21:07:13.358573 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250109 21:07:13.359242 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:13.345278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.12.126.126:38813)
0109 21:07:13.345526 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:13.345530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:13.347197 (+ 1667us) server_negotiation.cc:407] Connection header received
0109 21:07:13.347335 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:13.347342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:13.347402 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:13.347501 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:13.349383 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:13.350175 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:13.351060 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:13.351423 (+ 363us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:13.355093 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:13.355112 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:13.355118 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:13.355172 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:13.357047 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:13.357732 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:13.357734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:13.357736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:13.357776 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:13.358015 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:13.358018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:13.358022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:13.358150 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:13.358330 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:13.359027 (+ 697us) server_negotiation.cc:299] Negotiation successful
0109 21:07:13.359136 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
I20250109 21:07:13.359709 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:13.343299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:38813 (local address 127.12.126.67:43119)
0109 21:07:13.347589 (+ 4290us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:13.347632 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:13.348329 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:13.349746 (+ 1417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:13.349754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:13.349986 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:13.350633 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:13.350648 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:13.351733 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:13.351738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:13.352145 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:13.352154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:13.352373 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:13.352901 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:13.352925 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:13.354137 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:13.356174 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:13.356178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:13.356187 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:13.356422 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:13.356714 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:13.356717 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:13.356719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:13.356980 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:13.357805 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:13.357810 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:13.358119 (+ 309us) client_negotiation.cc:769] Sending connection context
0109 21:07:13.358416 (+ 297us) client_negotiation.cc:240] Negotiation successful
0109 21:07:13.358678 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4067,"thread_start_us":123,"threads_started":1}
I20250109 21:07:13.360831 26043 heartbeater.cc:346] Connected to a master server at 127.12.126.126:38813
I20250109 21:07:13.361058 26043 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:13.361646 26043 heartbeater.cc:510] Master 127.12.126.126:38813 requested a full tablet report, sending...
W20250109 21:07:13.363305 26043 heartbeater.cc:643] Failed to heartbeat to 127.12.126.126:38813 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20250109 21:07:13.363420 26043 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:13.363605 26043 heartbeater.cc:510] Master 127.12.126.126:38813 requested a full tablet report, sending...
I20250109 21:07:13.364470 26043 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:13.364698 26043 heartbeater.cc:510] Master 127.12.126.126:38813 requested a full tablet report, sending...
W20250109 21:07:13.365482 26043 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jan 09 21:07:13 dist-test-slave-g0zb krb5kdc[24559](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1736456833, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:13.907038 25775 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:13.907429 25775 heartbeater.cc:510] Master 127.12.126.126:38813 requested a full tablet report, sending...
I20250109 21:07:13.908704 25577 ts_manager.cc:194] Registered new tserver with Master: 056b12014b8b441187409a9fc324fdbb (127.12.126.65:35611)
I20250109 21:07:13.910502 25577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:44659
Jan 09 21:07:14 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456833, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for HTTP/127.12.126.84@KRBTEST.COM
I20250109 21:07:14.129866 25909 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:14.130228 25909 heartbeater.cc:510] Master 127.12.126.126:38813 requested a full tablet report, sending...
I20250109 21:07:14.131112 25577 ts_manager.cc:194] Registered new tserver with Master: 88acd837bcfa4de98dfd093ce8488660 (127.12.126.66:42541)
I20250109 21:07:14.131816 25577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:39851
Jan 09 21:07:14 dist-test-slave-g0zb krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456833, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for HTTP/127.12.126.84@KRBTEST.COM
I20250109 21:07:14.367056 26043 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:14.367399 26043 heartbeater.cc:510] Master 127.12.126.126:38813 requested a full tablet report, sending...
I20250109 21:07:14.368307 25577 ts_manager.cc:194] Registered new tserver with Master: 71bd1528498448dd9984969588aa4d00 (127.12.126.67:33881)
I20250109 21:07:14.368896 25577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:43119
I20250109 21:07:14.380041 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 21:07:14.407603 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:14.387263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33880 (local address 127.12.126.126:38813)
0109 21:07:14.395381 (+ 8118us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:14.395386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:14.395407 (+ 21us) server_negotiation.cc:407] Connection header received
0109 21:07:14.395462 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:14.395465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:14.395527 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:14.395642 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:14.399779 (+ 4137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.400650 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:14.401608 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.401810 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:14.403812 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:14.403837 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:14.403840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:14.403873 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:14.405799 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:14.406301 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:14.406305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:14.406307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:14.406362 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:14.406671 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:14.406675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:14.406677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:14.406880 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:14.407007 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:14.407241 (+ 234us) server_negotiation.cc:299] Negotiation successful
0109 21:07:14.407373 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7989,"thread_start_us":114,"threads_started":1}
I20250109 21:07:14.416785 25577 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:07:14.420228 25577 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:07:14.441937 26074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:14.434203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33868 (local address 127.12.126.65:35611)
0109 21:07:14.435742 (+ 1539us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:14.435749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:14.435772 (+ 23us) server_negotiation.cc:407] Connection header received
0109 21:07:14.435855 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:14.435861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:14.436052 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:14.436218 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:14.436727 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.438681 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:14.439978 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.440501 (+ 523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:14.440637 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:14.441662 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0109 21:07:14.441764 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1430,"thread_start_us":83,"threads_started":1}
I20250109 21:07:14.443316 26072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:14.434031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:35611 (local address 127.0.0.1:33868)
0109 21:07:14.435168 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:14.435181 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:14.435260 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:14.436242 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:14.436248 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:14.436280 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:14.436574 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:14.436584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.438868 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:14.438872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:14.439806 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:14.439818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.440706 (+ 888us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:14.440755 (+ 49us) client_negotiation.cc:769] Sending connection context
0109 21:07:14.443035 (+ 2280us) client_negotiation.cc:240] Negotiation successful
0109 21:07:14.443156 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"thread_start_us":88,"threads_started":1}
I20250109 21:07:14.446404 25697 tablet_service.cc:1467] Processing CreateTablet for tablet 00d86105a00f4b9e8cc74f71f09ce1ea (DEFAULT_TABLE table=test-table [id=4d6bed36899e473584f173ade98d9696]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:07:14.447461 25697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00d86105a00f4b9e8cc74f71f09ce1ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:14.450276 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:14.434337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:42541 (local address 127.0.0.1:38412)
0109 21:07:14.435434 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:14.435446 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:14.435510 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:14.438433 (+ 2923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:14.438437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:14.438457 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:14.438688 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:14.438701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.448957 (+ 10256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:14.448961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:14.449886 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:14.449898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.449987 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:14.450008 (+ 21us) client_negotiation.cc:769] Sending connection context
0109 21:07:14.450048 (+ 40us) client_negotiation.cc:240] Negotiation successful
0109 21:07:14.450113 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"thread_start_us":81,"threads_started":1}
I20250109 21:07:14.451684 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:14.434620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.12.126.66:42541)
0109 21:07:14.437172 (+ 2552us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:14.437178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:14.437199 (+ 21us) server_negotiation.cc:407] Connection header received
0109 21:07:14.437272 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:14.437277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:14.437433 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:14.445163 (+ 7730us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:14.447661 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.448802 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:14.450642 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.451271 (+ 629us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:14.451390 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:14.451461 (+ 71us) server_negotiation.cc:299] Negotiation successful
0109 21:07:14.451536 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2446,"thread_start_us":97,"threads_started":1}
I20250109 21:07:14.454366 25826 tablet_service.cc:1467] Processing CreateTablet for tablet 00d86105a00f4b9e8cc74f71f09ce1ea (DEFAULT_TABLE table=test-table [id=4d6bed36899e473584f173ade98d9696]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:07:14.455343 25826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00d86105a00f4b9e8cc74f71f09ce1ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:14.455560 26078 tablet_bootstrap.cc:492] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb: Bootstrap starting.
I20250109 21:07:14.458678 26078 tablet_bootstrap.cc:654] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb: Neither blocks nor log segments found. Creating new log.
I20250109 21:07:14.460150 26078 log.cc:826] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:14.462189 26078 tablet_bootstrap.cc:492] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb: No bootstrap required, opened a new log
I20250109 21:07:14.462385 26078 ts_tablet_manager.cc:1397] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20250109 21:07:14.466912 26078 raft_consensus.cc:357] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71bd1528498448dd9984969588aa4d00" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 33881 } } peers { permanent_uuid: "056b12014b8b441187409a9fc324fdbb" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35611 } } peers { permanent_uuid: "88acd837bcfa4de98dfd093ce8488660" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42541 } }
I20250109 21:07:14.467319 26078 raft_consensus.cc:383] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:14.467430 26078 raft_consensus.cc:738] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 056b12014b8b441187409a9fc324fdbb, State: Initialized, Role: FOLLOWER
I20250109 21:07:14.468029 26078 consensus_queue.cc:260] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71bd1528498448dd9984969588aa4d00" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 33881 } } peers { permanent_uuid: "056b12014b8b441187409a9fc324fdbb" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35611 } } peers { permanent_uuid: "88acd837bcfa4de98dfd093ce8488660" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42541 } }
I20250109 21:07:14.469396 26078 ts_tablet_manager.cc:1428] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250109 21:07:14.469838 25775 heartbeater.cc:502] Master 127.12.126.126:38813 was elected leader, sending a full tablet report...
I20250109 21:07:14.471796 26075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:14.433287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:33881 (local address 127.0.0.1:32840)
0109 21:07:14.434740 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:14.434779 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:14.434929 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:14.444607 (+ 9678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:14.444611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:14.444635 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:14.445654 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:14.445664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.470355 (+ 24691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:14.470360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:14.471390 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:14.471405 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.471507 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:14.471531 (+ 24us) client_negotiation.cc:769] Sending connection context
0109 21:07:14.471572 (+ 41us) client_negotiation.cc:240] Negotiation successful
0109 21:07:14.471645 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1285,"thread_start_us":113,"threads_started":1}
I20250109 21:07:14.473192 26077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:14.434867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.12.126.67:33881)
0109 21:07:14.436518 (+ 1651us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:14.436524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:14.436546 (+ 22us) server_negotiation.cc:407] Connection header received
0109 21:07:14.436622 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:14.436627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:14.436791 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:14.444638 (+ 7847us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:14.445806 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.470141 (+ 24335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:14.472152 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.472738 (+ 586us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:14.472867 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:14.472944 (+ 77us) server_negotiation.cc:299] Negotiation successful
0109 21:07:14.473031 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1553,"thread_start_us":89,"threads_started":1}
I20250109 21:07:14.476074 25976 tablet_service.cc:1467] Processing CreateTablet for tablet 00d86105a00f4b9e8cc74f71f09ce1ea (DEFAULT_TABLE table=test-table [id=4d6bed36899e473584f173ade98d9696]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:07:14.477104 25976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00d86105a00f4b9e8cc74f71f09ce1ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:14.490705 26082 tablet_bootstrap.cc:492] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00: Bootstrap starting.
I20250109 21:07:14.493312 26082 tablet_bootstrap.cc:654] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00: Neither blocks nor log segments found. Creating new log.
I20250109 21:07:14.494719 26082 log.cc:826] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:14.495254 26080 tablet_bootstrap.cc:492] T 00d86105a00f4b9e8cc74f71f09ce1ea P 88acd837bcfa4de98dfd093ce8488660: Bootstrap starting.
I20250109 21:07:14.497654 26080 tablet_bootstrap.cc:654] T 00d86105a00f4b9e8cc74f71f09ce1ea P 88acd837bcfa4de98dfd093ce8488660: Neither blocks nor log segments found. Creating new log.
I20250109 21:07:14.498904 26080 log.cc:826] T 00d86105a00f4b9e8cc74f71f09ce1ea P 88acd837bcfa4de98dfd093ce8488660: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:14.500981 26082 tablet_bootstrap.cc:492] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00: No bootstrap required, opened a new log
I20250109 21:07:14.501281 26082 ts_tablet_manager.cc:1397] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20250109 21:07:14.505614 26080 tablet_bootstrap.cc:492] T 00d86105a00f4b9e8cc74f71f09ce1ea P 88acd837bcfa4de98dfd093ce8488660: No bootstrap required, opened a new log
I20250109 21:07:14.505820 26080 ts_tablet_manager.cc:1397] T 00d86105a00f4b9e8cc74f71f09ce1ea P 88acd837bcfa4de98dfd093ce8488660: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20250109 21:07:14.507591 26082 raft_consensus.cc:357] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71bd1528498448dd9984969588aa4d00" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 33881 } } peers { permanent_uuid: "056b12014b8b441187409a9fc324fdbb" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35611 } } peers { permanent_uuid: "88acd837bcfa4de98dfd093ce8488660" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42541 } }
I20250109 21:07:14.508003 26082 raft_consensus.cc:383] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:14.508109 26082 raft_consensus.cc:738] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71bd1528498448dd9984969588aa4d00, State: Initialized, Role: FOLLOWER
I20250109 21:07:14.508668 26082 consensus_queue.cc:260] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71bd1528498448dd9984969588aa4d00" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 33881 } } peers { permanent_uuid: "056b12014b8b441187409a9fc324fdbb" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35611 } } peers { permanent_uuid: "88acd837bcfa4de98dfd093ce8488660" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42541 } }
I20250109 21:07:14.509207 26080 raft_consensus.cc:357] T 00d86105a00f4b9e8cc74f71f09ce1ea P 88acd837bcfa4de98dfd093ce8488660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71bd1528498448dd9984969588aa4d00" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 33881 } } peers { permanent_uuid: "056b12014b8b441187409a9fc324fdbb" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35611 } } peers { permanent_uuid: "88acd837bcfa4de98dfd093ce8488660" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42541 } }
I20250109 21:07:14.509443 26080 raft_consensus.cc:383] T 00d86105a00f4b9e8cc74f71f09ce1ea P 88acd837bcfa4de98dfd093ce8488660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:14.509510 26080 raft_consensus.cc:738] T 00d86105a00f4b9e8cc74f71f09ce1ea P 88acd837bcfa4de98dfd093ce8488660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88acd837bcfa4de98dfd093ce8488660, State: Initialized, Role: FOLLOWER
I20250109 21:07:14.509887 26082 ts_tablet_manager.cc:1428] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250109 21:07:14.509968 26080 consensus_queue.cc:260] T 00d86105a00f4b9e8cc74f71f09ce1ea P 88acd837bcfa4de98dfd093ce8488660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71bd1528498448dd9984969588aa4d00" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 33881 } } peers { permanent_uuid: "056b12014b8b441187409a9fc324fdbb" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35611 } } peers { permanent_uuid: "88acd837bcfa4de98dfd093ce8488660" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42541 } }
I20250109 21:07:14.510407 26043 heartbeater.cc:502] Master 127.12.126.126:38813 was elected leader, sending a full tablet report...
I20250109 21:07:14.512041 26080 ts_tablet_manager.cc:1428] T 00d86105a00f4b9e8cc74f71f09ce1ea P 88acd837bcfa4de98dfd093ce8488660: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250109 21:07:14.512445 25909 heartbeater.cc:502] Master 127.12.126.126:38813 was elected leader, sending a full tablet report...
I20250109 21:07:14.524200 26081 raft_consensus.cc:491] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:07:14.524423 26081 raft_consensus.cc:513] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71bd1528498448dd9984969588aa4d00" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 33881 } } peers { permanent_uuid: "056b12014b8b441187409a9fc324fdbb" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35611 } } peers { permanent_uuid: "88acd837bcfa4de98dfd093ce8488660" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42541 } }
I20250109 21:07:14.525846 26081 leader_election.cc:290] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71bd1528498448dd9984969588aa4d00 (127.12.126.67:33881), 88acd837bcfa4de98dfd093ce8488660 (127.12.126.66:42541)
I20250109 21:07:14.530926 26087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:14.526085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:33881 (local address 127.12.126.65:60729)
0109 21:07:14.526416 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:14.526751 (+ 335us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:14.526898 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:14.527364 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:14.527368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:14.527389 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:14.527711 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:14.527719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.528815 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:14.528818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:14.529807 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:14.529817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.530572 (+ 755us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:14.530591 (+ 19us) client_negotiation.cc:769] Sending connection context
0109 21:07:14.530667 (+ 76us) client_negotiation.cc:240] Negotiation successful
0109 21:07:14.530729 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":122,"threads_started":1}
I20250109 21:07:14.531342 26077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:14.526878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:60729 (local address 127.12.126.67:33881)
0109 21:07:14.527023 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:14.527028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:14.527045 (+ 17us) server_negotiation.cc:407] Connection header received
0109 21:07:14.527097 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:14.527101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:14.527151 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:14.527244 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:14.527853 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.528668 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:14.529954 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.530459 (+ 505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:14.530505 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:14.531128 (+ 623us) server_negotiation.cc:299] Negotiation successful
0109 21:07:14.531208 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250109 21:07:14.531960 25979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d86105a00f4b9e8cc74f71f09ce1ea" candidate_uuid: "056b12014b8b441187409a9fc324fdbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71bd1528498448dd9984969588aa4d00" is_pre_election: true
I20250109 21:07:14.532294 25979 raft_consensus.cc:2463] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 056b12014b8b441187409a9fc324fdbb in term 0.
I20250109 21:07:14.532783 25662 leader_election.cc:304] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056b12014b8b441187409a9fc324fdbb, 71bd1528498448dd9984969588aa4d00; no voters:
I20250109 21:07:14.533071 26081 raft_consensus.cc:2798] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:07:14.533154 26081 raft_consensus.cc:491] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:07:14.533193 26081 raft_consensus.cc:3054] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:14.534432 26081 raft_consensus.cc:513] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71bd1528498448dd9984969588aa4d00" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 33881 } } peers { permanent_uuid: "056b12014b8b441187409a9fc324fdbb" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35611 } } peers { permanent_uuid: "88acd837bcfa4de98dfd093ce8488660" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42541 } }
I20250109 21:07:14.534873 26081 leader_election.cc:290] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [CANDIDATE]: Term 1 election: Requested vote from peers 71bd1528498448dd9984969588aa4d00 (127.12.126.67:33881), 88acd837bcfa4de98dfd093ce8488660 (127.12.126.66:42541)
I20250109 21:07:14.535362 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:14.526591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:42851 (local address 127.12.126.66:42541)
0109 21:07:14.526715 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:14.526719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:14.527192 (+ 473us) server_negotiation.cc:407] Connection header received
0109 21:07:14.527387 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:14.527390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:14.527444 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:14.527496 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:14.532270 (+ 4774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.532985 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:14.534315 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.534763 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:14.534832 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:14.535158 (+ 326us) server_negotiation.cc:299] Negotiation successful
0109 21:07:14.535246 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250109 21:07:14.535355 25979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d86105a00f4b9e8cc74f71f09ce1ea" candidate_uuid: "056b12014b8b441187409a9fc324fdbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71bd1528498448dd9984969588aa4d00"
I20250109 21:07:14.535485 25979 raft_consensus.cc:3054] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:14.535876 26088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:14.526509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:42541 (local address 127.12.126.65:42851)
0109 21:07:14.527120 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:14.527133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:14.527259 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:14.527630 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:14.527633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:14.527651 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:14.528447 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:14.528460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.533349 (+ 4889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:14.533355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:14.534152 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:14.534164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.534996 (+ 832us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:14.535016 (+ 20us) client_negotiation.cc:769] Sending connection context
0109 21:07:14.535655 (+ 639us) client_negotiation.cc:240] Negotiation successful
0109 21:07:14.535734 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":81,"threads_started":1}
I20250109 21:07:14.536705 25979 raft_consensus.cc:2463] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 056b12014b8b441187409a9fc324fdbb in term 1.
I20250109 21:07:14.536762 25846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d86105a00f4b9e8cc74f71f09ce1ea" candidate_uuid: "056b12014b8b441187409a9fc324fdbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88acd837bcfa4de98dfd093ce8488660" is_pre_election: true
I20250109 21:07:14.537067 25662 leader_election.cc:304] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056b12014b8b441187409a9fc324fdbb, 71bd1528498448dd9984969588aa4d00; no voters:
I20250109 21:07:14.537073 25846 raft_consensus.cc:2463] T 00d86105a00f4b9e8cc74f71f09ce1ea P 88acd837bcfa4de98dfd093ce8488660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 056b12014b8b441187409a9fc324fdbb in term 0.
I20250109 21:07:14.537252 26081 raft_consensus.cc:2798] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:07:14.537434 25849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d86105a00f4b9e8cc74f71f09ce1ea" candidate_uuid: "056b12014b8b441187409a9fc324fdbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88acd837bcfa4de98dfd093ce8488660"
I20250109 21:07:14.537477 26081 raft_consensus.cc:695] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [term 1 LEADER]: Becoming Leader. State: Replica: 056b12014b8b441187409a9fc324fdbb, State: Running, Role: LEADER
I20250109 21:07:14.537551 25849 raft_consensus.cc:3054] T 00d86105a00f4b9e8cc74f71f09ce1ea P 88acd837bcfa4de98dfd093ce8488660 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:14.537827 26081 consensus_queue.cc:237] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71bd1528498448dd9984969588aa4d00" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 33881 } } peers { permanent_uuid: "056b12014b8b441187409a9fc324fdbb" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35611 } } peers { permanent_uuid: "88acd837bcfa4de98dfd093ce8488660" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42541 } }
I20250109 21:07:14.538933 25849 raft_consensus.cc:2463] T 00d86105a00f4b9e8cc74f71f09ce1ea P 88acd837bcfa4de98dfd093ce8488660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 056b12014b8b441187409a9fc324fdbb in term 1.
I20250109 21:07:14.541291 25576 catalog_manager.cc:5526] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 056b12014b8b441187409a9fc324fdbb (127.12.126.65). New cstate: current_term: 1 leader_uuid: "056b12014b8b441187409a9fc324fdbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71bd1528498448dd9984969588aa4d00" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 33881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "056b12014b8b441187409a9fc324fdbb" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 35611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88acd837bcfa4de98dfd093ce8488660" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42541 } health_report { overall_health: UNKNOWN } } }
I20250109 21:07:14.571372 26074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:14.567353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33884 (local address 127.12.126.65:35611)
0109 21:07:14.567490 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:14.567495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:14.567511 (+ 16us) server_negotiation.cc:407] Connection header received
0109 21:07:14.567610 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:14.567613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:14.567686 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:14.567814 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:07:14.568358 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.568830 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:14.569734 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:14.569920 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:14.570013 (+ 93us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:07:14.570587 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:07:14.570713 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:14.571163 (+ 450us) server_negotiation.cc:299] Negotiation successful
0109 21:07:14.571221 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250109 21:07:14.578713 25849 raft_consensus.cc:1270] T 00d86105a00f4b9e8cc74f71f09ce1ea P 88acd837bcfa4de98dfd093ce8488660 [term 1 FOLLOWER]: Refusing update from remote peer 056b12014b8b441187409a9fc324fdbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:07:14.579749 26089 consensus_queue.cc:1035] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "88acd837bcfa4de98dfd093ce8488660" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:07:14.587178 25979 raft_consensus.cc:1270] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00 [term 1 FOLLOWER]: Refusing update from remote peer 056b12014b8b441187409a9fc324fdbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:07:14.587879 26089 consensus_queue.cc:1035] T 00d86105a00f4b9e8cc74f71f09ce1ea P 056b12014b8b441187409a9fc324fdbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71bd1528498448dd9984969588aa4d00" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 33881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 21:07:14.593575 26044 tablet.cc:2367] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:07:14.608924 25576 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33880:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:07:14.609108 25576 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33880:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:07:14.612505 25576 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 3507bd54644843a5991ce63e234d1925: Sending DeleteTablet for 3 replicas of tablet 00d86105a00f4b9e8cc74f71f09ce1ea
I20250109 21:07:14.613317 25697 tablet_service.cc:1514] Processing DeleteTablet for tablet 00d86105a00f4b9e8cc74f71f09ce1ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:07:14 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:33868
I20250109 21:07:14.613379 26092 mvcc.cc:204] Tried to move back new op lower bound from 7112527194424705024 to 7112527194269818880. Current Snapshot: MvccSnapshot[applied={T|T < 7112527194424705024 or (T in {7112527194424705024})}]
I20250109 21:07:14.613879 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 25615
I20250109 21:07:14.613847 26093 mvcc.cc:204] Tried to move back new op lower bound from 7112527194424705024 to 7112527194269818880. Current Snapshot: MvccSnapshot[applied={T|T < 7112527194424705024 or (T in {7112527194424705024})}]
I20250109 21:07:14.613996 25976 tablet_service.cc:1514] Processing DeleteTablet for tablet 00d86105a00f4b9e8cc74f71f09ce1ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:07:14 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:32840
I20250109 21:07:14.616822 25826 tablet_service.cc:1514] Processing DeleteTablet for tablet 00d86105a00f4b9e8cc74f71f09ce1ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:07:14 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:38412
I20250109 21:07:14.617326 26101 tablet_replica.cc:331] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00: stopping tablet replica
I20250109 21:07:14.617586 26101 raft_consensus.cc:2238] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:07:14.617830 26101 pending_rounds.cc:62] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00: Trying to abort 2 pending ops.
I20250109 21:07:14.617908 26101 pending_rounds.cc:69] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7112527194424705024 op_type: WRITE_OP write_request { tablet_id: "00d86105a00f4b9e8cc74f71f09ce1ea" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\256\367\200\274\006\"8\n\ntest-admin\022*\n 4d6bed36899e473584f173ade98d9696\020\001\030\001 \001(\001" signature: "<redacted>""\"\260;G\256HK\257\301\037@\330\270i:\303\377\356s\376\215\327\203W\\\222\221f\034\254\215\005\271M9\302\252\246\020\203\201\333\177~\232\026/=\025\340\304\247\302\247$\024 7\315o\304\032=\234" signing_key_seq_num: 0 } } request_id { client_id: "0160e386904b45ce9b0f9fce708b3c74" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20250109 21:07:14.618199 26101 pending_rounds.cc:69] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112527194269818880 op_type: NO_OP noop_request { }
I20250109 21:07:14.618283 26101 raft_consensus.cc:2883] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250109 21:07:14.618369 26101 raft_consensus.cc:2267] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:07:14.618695 26102 tablet_replica.cc:331] T 00d86105a00f4b9e8cc74f71f09ce1ea P 88acd837bcfa4de98dfd093ce8488660: stopping tablet replica
I20250109 21:07:14.618960 26102 raft_consensus.cc:2238] T 00d86105a00f4b9e8cc74f71f09ce1ea P 88acd837bcfa4de98dfd093ce8488660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:07:14.619261 26102 raft_consensus.cc:2267] T 00d86105a00f4b9e8cc74f71f09ce1ea P 88acd837bcfa4de98dfd093ce8488660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:07:14.623800 26101 ts_tablet_manager.cc:1905] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 21:07:14.624591 25932 connection.cc:537] server connection from 127.12.126.65:60729 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 21:07:14.624994 25561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.126.65:35611 (error 108)
I20250109 21:07:14.626623 26075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:14.626235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:35611 (local address 127.0.0.1:33894)
0109 21:07:14.626449 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:14.626523 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.65:35611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20250109 21:07:14.626636 26101 ts_tablet_manager.cc:1918] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:07:14.626715 26101 log.cc:1198] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/00d86105a00f4b9e8cc74f71f09ce1ea
W20250109 21:07:14.626822 25561 catalog_manager.cc:4615] TS 056b12014b8b441187409a9fc324fdbb (127.12.126.65:35611): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00d86105a00f4b9e8cc74f71f09ce1ea: Network error: Client connection negotiation failed: client connection to 127.12.126.65:35611: connect: Connection refused (error 111)
I20250109 21:07:14.627121 26101 ts_tablet_manager.cc:1939] T 00d86105a00f4b9e8cc74f71f09ce1ea P 71bd1528498448dd9984969588aa4d00: Deleting consensus metadata
I20250109 21:07:14.627161 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 25779
I20250109 21:07:14.634439 26075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:14.634117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:42541 (local address 127.0.0.1:38428)
0109 21:07:14.634301 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:14.634351 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.66:42541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20250109 21:07:14.635262 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 25911
I20250109 21:07:14.637459 25561 catalog_manager.cc:4872] TS 71bd1528498448dd9984969588aa4d00 (127.12.126.67:33881): tablet 00d86105a00f4b9e8cc74f71f09ce1ea (table test-table [id=4d6bed36899e473584f173ade98d9696]) successfully deleted
I20250109 21:07:14.643800 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 25544
I20250109 21:07:14.650712 12793 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20250109 21:07:14.985065 12793 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20250109 21:07:14.985188 12793 mini_ranger.cc:66] Stopping Ranger...
I20250109 21:07:15.331068 12793 mini_ranger.cc:68] Stopped Ranger
2025-01-09 21:07:15.331 UTC [24589] LOG: received smart shutdown request
2025-01-09 21:07:15.334 UTC [24589] LOG: background worker "logical replication launcher" (PID 24597) exited with exit code 1
2025-01-09 21:07:15.335 UTC [24592] LOG: shutting down
2025-01-09 21:07:15.337 UTC [24592] LOG: checkpoint starting: shutdown immediate
2025-01-09 21:07:16.216 UTC [24592] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.034 s, sync=0.840 s, total=0.881 s; sync files=1108, longest=0.005 s, average=0.001 s; distance=12074 kB, estimate=12074 kB; lsn=0/2045118, redo lsn=0/2045118
2025-01-09 21:07:16.224 UTC [24589] LOG: database system is shut down
2025-01-09T21:07:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46463 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:07:16 dist-test-slave-g0zb krb5kdc[26112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:07:16 dist-test-slave-g0zb krb5kdc[26112](info): set up 2 sockets
Jan 09 21:07:16 dist-test-slave-g0zb krb5kdc[26112](info): commencing operation
krb5kdc: starting...
W20250109 21:07:17.375351 12793 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:07:17 dist-test-slave-g0zb krb5kdc[26112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:07:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:07:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:07:17.487242 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:40053
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:41127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:40053,127.12.126.125:36789,127.12.126.124:46765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:07:17.592125 26125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:17.592353 26125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:17.592437 26125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:17.595474 26125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:07:17.595531 26125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:17.595568 26125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:17.595611 26125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:07:17.595654 26125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:07:17.599766 26125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:41127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:40053,127.12.126.125:36789,127.12.126.124:46765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:40053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:17.600750 26125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:17.601540 26125 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:17.608754 26130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:17.608853 26133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:17.609710 26131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:17.610615 26125 server_base.cc:1034] running on GCE node
I20250109 21:07:17.611140 26125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:17.612020 26125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:17.613178 26125 hybrid_clock.cc:648] HybridClock initialized: now 1736456837613162 us; error 31 us; skew 500 ppm
Jan 09 21:07:17 dist-test-slave-g0zb krb5kdc[26112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456837, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:17.615656 26125 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:07:17.616484 26125 webserver.cc:458] Webserver started at http://127.12.126.126:44771/ using document root <none> and password file <none>
I20250109 21:07:17.617002 26125 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:17.617071 26125 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:17.617254 26125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:07:17.619024 26125 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "c7ebf6b2be1e46b087668033126a9de2"
format_stamp: "Formatted at 2025-01-09 21:07:17 on dist-test-slave-g0zb"
server_key: "55bfef5e1eee2029736a1bdf4c38f884"
server_key_iv: "aeec02934fdac07ca2fd7cdfd1ad286f"
server_key_version: "encryptionkey@0"
I20250109 21:07:17.619494 26125 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "c7ebf6b2be1e46b087668033126a9de2"
format_stamp: "Formatted at 2025-01-09 21:07:17 on dist-test-slave-g0zb"
server_key: "55bfef5e1eee2029736a1bdf4c38f884"
server_key_iv: "aeec02934fdac07ca2fd7cdfd1ad286f"
server_key_version: "encryptionkey@0"
I20250109 21:07:17.623200 26125 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:07:17.625358 26140 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:17.626353 26125 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:07:17.626461 26125 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "c7ebf6b2be1e46b087668033126a9de2"
format_stamp: "Formatted at 2025-01-09 21:07:17 on dist-test-slave-g0zb"
server_key: "55bfef5e1eee2029736a1bdf4c38f884"
server_key_iv: "aeec02934fdac07ca2fd7cdfd1ad286f"
server_key_version: "encryptionkey@0"
I20250109 21:07:17.626543 26125 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:17.644657 26125 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:17.647826 26125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:17.647993 26125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:17.654407 26125 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:40053
I20250109 21:07:17.654424 26192 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:40053 every 8 connection(s)
I20250109 21:07:17.655421 26125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:07:17.658102 26193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:17.661332 26193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } has no permanent_uuid. Determining permanent_uuid...
I20250109 21:07:17.662324 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 26125
I20250109 21:07:17.662441 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:07:17.662699 12793 external_mini_cluster.cc:1453] Setting key 7f95c57434c40a03594031f56612d2ae
Jan 09 21:07:17 dist-test-slave-g0zb krb5kdc[26112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456837, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.126.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.125@KRBTEST.COM" created.
I20250109 21:07:17.675192 26197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.664301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46690 (local address 127.12.126.126:40053)
0109 21:07:17.664653 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:17.664657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:17.664689 (+ 32us) server_negotiation.cc:407] Connection header received
0109 21:07:17.665075 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:17.665079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:17.665227 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:17.665371 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:17.666595 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:17.667361 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:17.668266 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:17.668417 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:17.670962 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:17.670981 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:17.670989 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:17.671011 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:17.672639 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:17.673062 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:17.673066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:17.673067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:17.673121 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:17.673406 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:17.673409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:17.673410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:17.673822 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:17.673961 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:17.674308 (+ 347us) server_negotiation.cc:299] Negotiation successful
0109 21:07:17.674541 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":276,"thread_start_us":91,"threads_started":1}
I20250109 21:07:17.675199 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.663422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:40053 (local address 127.0.0.1:46690)
0109 21:07:17.663886 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:17.663919 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:17.664761 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:17.665420 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:17.665428 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:17.665984 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:17.666444 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:17.666464 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:17.667476 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:17.667482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:17.668149 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:17.668163 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:17.668399 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:17.669032 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:17.669071 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:17.670816 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:17.672733 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:17.672739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:17.672747 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:17.672943 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:17.673200 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:17.673203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:17.673205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:17.673312 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:17.673930 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:17.673935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:17.674063 (+ 128us) client_negotiation.cc:769] Sending connection context
0109 21:07:17.674208 (+ 145us) client_negotiation.cc:240] Negotiation successful
0109 21:07:17.674424 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":129,"threads_started":1}
I20250109 21:07:17.677376 26193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } has no permanent_uuid. Determining permanent_uuid...
I20250109 21:07:17.678179 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.677858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:36789 (local address 127.0.0.1:35950)
0109 21:07:17.678009 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:17.678093 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.125:36789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20250109 21:07:17.678360 26142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.126.125:36789: connect: Connection refused (error 111)
I20250109 21:07:17.679196 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.678933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:36789 (local address 127.0.0.1:35956)
0109 21:07:17.679073 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:17.679130 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.125:36789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20250109 21:07:17.679356 26193 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.125:36789: Network error: Client connection negotiation failed: client connection to 127.12.126.125:36789: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.125.keytab.
Entry for principal kudu/127.12.126.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.125.keytab.
I20250109 21:07:17.688201 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.125:36789
--webserver_interface=127.12.126.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.125.keytab
--principal=kudu/127.12.126.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:41127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:40053,127.12.126.125:36789,127.12.126.124:46765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:07:17.728551 26193 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } attempt: 1
I20250109 21:07:17.729451 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.729083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:36789 (local address 127.0.0.1:35958)
0109 21:07:17.729277 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:17.729352 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.125:36789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20250109 21:07:17.730185 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.729939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:36789 (local address 127.0.0.1:35970)
0109 21:07:17.730045 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:17.730117 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.125:36789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20250109 21:07:17.730383 26193 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.125:36789: Network error: Client connection negotiation failed: client connection to 127.12.126.125:36789: connect: Connection refused (error 111)
W20250109 21:07:17.791256 26200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:17.791474 26200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:17.791551 26200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:17.794591 26200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:07:17.794651 26200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:17.794690 26200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:17.794734 26200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:07:17.794777 26200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:07:17.798632 26193 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } attempt: 2
I20250109 21:07:17.798940 26200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:41127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:40053,127.12.126.125:36789,127.12.126.124:46765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.125.keytab
--principal=kudu/127.12.126.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.125:36789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.126.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:17.799497 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.799164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:36789 (local address 127.0.0.1:35984)
0109 21:07:17.799342 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:17.799409 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.125:36789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20250109 21:07:17.800127 26200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:17.800190 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.799963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:36789 (local address 127.0.0.1:35986)
0109 21:07:17.800066 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:17.800117 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.125:36789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20250109 21:07:17.800314 26193 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.125:36789: Network error: Client connection negotiation failed: client connection to 127.12.126.125:36789: connect: Connection refused (error 111)
I20250109 21:07:17.800926 26200 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:17.807533 26208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:17.808252 26205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:17.808873 26206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:17.809958 26200 server_base.cc:1034] running on GCE node
I20250109 21:07:17.810355 26200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:17.811167 26200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:17.812316 26200 hybrid_clock.cc:648] HybridClock initialized: now 1736456837812305 us; error 23 us; skew 500 ppm
Jan 09 21:07:17 dist-test-slave-g0zb krb5kdc[26112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456837, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:17.814731 26200 init.cc:377] Logged in from keytab as kudu/127.12.126.125@KRBTEST.COM (short username kudu)
I20250109 21:07:17.815620 26200 webserver.cc:458] Webserver started at http://127.12.126.125:45253/ using document root <none> and password file <none>
I20250109 21:07:17.816135 26200 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:17.816187 26200 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:17.816370 26200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:07:17.818028 26200 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-1/data/instance:
uuid: "31d23ea953dd4903ba7dd47a8220c058"
format_stamp: "Formatted at 2025-01-09 21:07:17 on dist-test-slave-g0zb"
server_key: "b69f02c0c4dea5d54e247be59aaee07e"
server_key_iv: "22f8be0f54545e18643be28195704c9a"
server_key_version: "encryptionkey@0"
I20250109 21:07:17.818480 26200 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-1/wal/instance:
uuid: "31d23ea953dd4903ba7dd47a8220c058"
format_stamp: "Formatted at 2025-01-09 21:07:17 on dist-test-slave-g0zb"
server_key: "b69f02c0c4dea5d54e247be59aaee07e"
server_key_iv: "22f8be0f54545e18643be28195704c9a"
server_key_version: "encryptionkey@0"
I20250109 21:07:17.821612 26200 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:07:17.823621 26215 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:17.824556 26200 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:07:17.824652 26200 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-1/wal
uuid: "31d23ea953dd4903ba7dd47a8220c058"
format_stamp: "Formatted at 2025-01-09 21:07:17 on dist-test-slave-g0zb"
server_key: "b69f02c0c4dea5d54e247be59aaee07e"
server_key_iv: "22f8be0f54545e18643be28195704c9a"
server_key_version: "encryptionkey@0"
I20250109 21:07:17.824738 26200 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:17.848489 26200 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:17.851408 26200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:17.851585 26200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:17.858059 26200 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.125:36789
I20250109 21:07:17.858078 26267 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.125:36789 every 8 connection(s)
I20250109 21:07:17.859020 26200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-1/data/info.pb
I20250109 21:07:17.861665 26268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:17.862978 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 26200
I20250109 21:07:17.863066 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-1/wal/instance
I20250109 21:07:17.863296 12793 external_mini_cluster.cc:1453] Setting key 9cb528eaeef48fff640e51cfb084ca54
I20250109 21:07:17.865005 26268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.126.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.124@KRBTEST.COM" created.
Jan 09 21:07:17 dist-test-slave-g0zb krb5kdc[26112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456837, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.125@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:07:17.878072 26197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.866963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46696 (local address 127.12.126.126:40053)
0109 21:07:17.867152 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:17.867159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:17.867844 (+ 685us) server_negotiation.cc:407] Connection header received
0109 21:07:17.868959 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:17.868961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:17.869018 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:17.869093 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:17.870773 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:17.871333 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:17.871916 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:17.872042 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:17.874776 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:17.874794 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:17.874819 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:17.874851 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:17.876340 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:17.876785 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:17.876788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:17.876790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:17.876831 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:17.877078 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:17.877081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:17.877083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:17.877229 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:17.877317 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:17.877768 (+ 451us) server_negotiation.cc:299] Negotiation successful
0109 21:07:17.877884 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20250109 21:07:17.878621 26271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.867167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:40053 (local address 127.0.0.1:46696)
0109 21:07:17.867682 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:17.867730 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:17.868733 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:17.869270 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:17.869283 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:17.869813 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:17.870640 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:17.870660 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:17.871440 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:17.871443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:17.871829 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:17.871836 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:17.872000 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:17.872574 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:17.872596 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:17.874625 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:17.876452 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:17.876457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:17.876467 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:17.876688 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:17.876910 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:17.876915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:17.876917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:17.876996 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:17.877327 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:17.877334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:17.877552 (+ 218us) client_negotiation.cc:769] Sending connection context
0109 21:07:17.877722 (+ 170us) client_negotiation.cc:240] Negotiation successful
0109 21:07:17.877965 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":107,"threads_started":1}
I20250109 21:07:17.879973 26268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } has no permanent_uuid. Determining permanent_uuid...
Jan 09 21:07:17 dist-test-slave-g0zb krb5kdc[26112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456837, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.125@KRBTEST.COM for kudu/127.12.126.125@KRBTEST.COM
I20250109 21:07:17.887512 26271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.880404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:36789 (local address 127.0.0.1:36000)
0109 21:07:17.880613 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:17.880629 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:17.880725 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:17.881102 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:17.881105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:17.881280 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:17.881425 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:17.881430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:17.882413 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:17.882416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:17.882739 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:17.882745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:17.882862 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:17.883328 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:17.883339 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:17.884484 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:17.886173 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:17.886176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:17.886178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:17.886331 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:17.886553 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:17.886556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:17.886558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:17.886595 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:17.887019 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:17.887022 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:17.887094 (+ 72us) client_negotiation.cc:769] Sending connection context
0109 21:07:17.887233 (+ 139us) client_negotiation.cc:240] Negotiation successful
0109 21:07:17.887345 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20250109 21:07:17.887565 26273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.880506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36000 (local address 127.12.126.125:36789)
0109 21:07:17.880807 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:17.880813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:17.880833 (+ 20us) server_negotiation.cc:407] Connection header received
0109 21:07:17.880868 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:17.880871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:17.880990 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:17.881104 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:17.881525 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:17.882322 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:17.882843 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:17.883074 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:17.884586 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:17.884608 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:17.884617 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:17.884636 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:17.886072 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:17.886431 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:17.886435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:17.886436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:17.886479 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:17.886664 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:17.886669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:17.886673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:17.886920 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:17.887046 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:17.887319 (+ 273us) server_negotiation.cc:299] Negotiation successful
0109 21:07:17.887432 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":93,"threads_started":1}
I20250109 21:07:17.889153 26268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } has no permanent_uuid. Determining permanent_uuid...
I20250109 21:07:17.889741 26271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.889533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:46765 (local address 127.0.0.1:38128)
0109 21:07:17.889630 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:17.889688 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20250109 21:07:17.889892 26218 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.124.keytab.
Entry for principal kudu/127.12.126.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.124.keytab.
I20250109 21:07:17.890595 26271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.890425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:46765 (local address 127.0.0.1:38138)
0109 21:07:17.890506 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:17.890546 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20250109 21:07:17.890725 26268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.124:46765: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
I20250109 21:07:17.891203 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.124:46765
--webserver_interface=127.12.126.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.124.keytab
--principal=kudu/127.12.126.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:41127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:40053,127.12.126.125:36789,127.12.126.124:46765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:07:17.891469 26193 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } attempt: 3
Jan 09 21:07:17 dist-test-slave-g0zb krb5kdc[26112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456837, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for kudu/127.12.126.125@KRBTEST.COM
I20250109 21:07:17.899421 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.891843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:36789 (local address 127.0.0.1:36002)
0109 21:07:17.891949 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:17.891963 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:17.892035 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:17.892249 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:17.892252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:17.892461 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:17.892741 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:17.892751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:17.893516 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:17.893518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:17.893851 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:17.893858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:17.893943 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:17.894415 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:17.894428 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:17.896056 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:17.898077 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:17.898080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:17.898081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:17.898234 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:17.898467 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:17.898469 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:17.898471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:17.898506 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:17.898948 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:17.898950 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:17.899023 (+ 73us) client_negotiation.cc:769] Sending connection context
0109 21:07:17.899119 (+ 96us) client_negotiation.cc:240] Negotiation successful
0109 21:07:17.899246 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20250109 21:07:17.899421 26273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.891895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36002 (local address 127.12.126.125:36789)
0109 21:07:17.892036 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:17.892041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:17.892057 (+ 16us) server_negotiation.cc:407] Connection header received
0109 21:07:17.892096 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:17.892102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:17.892150 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:17.892235 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:17.892855 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:17.893394 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:17.893949 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:17.894184 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:17.896177 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:17.896197 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:17.896204 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:17.896236 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:17.897955 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:17.898323 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:17.898329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:17.898334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:17.898388 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:17.898566 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:17.898571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:17.898575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:17.898775 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:17.898958 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:17.899190 (+ 232us) server_negotiation.cc:299] Negotiation successful
0109 21:07:17.899291 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250109 21:07:17.900295 26193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } has no permanent_uuid. Determining permanent_uuid...
I20250109 21:07:17.900960 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.900737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:46765 (local address 127.0.0.1:38150)
0109 21:07:17.900863 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:17.900906 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20250109 21:07:17.901612 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.901407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:46765 (local address 127.0.0.1:38160)
0109 21:07:17.901531 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:17.901568 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20250109 21:07:17.901759 26193 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.124:46765: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
I20250109 21:07:17.932947 26193 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } attempt: 1
I20250109 21:07:17.933759 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.933484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:46765 (local address 127.0.0.1:38172)
0109 21:07:17.933626 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:17.933681 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20250109 21:07:17.934463 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.934270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:46765 (local address 127.0.0.1:38174)
0109 21:07:17.934371 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:17.934412 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20250109 21:07:17.934579 26193 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.124:46765: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
I20250109 21:07:17.939922 26268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } attempt: 1
I20250109 21:07:17.940551 26271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.940314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:46765 (local address 127.0.0.1:38190)
0109 21:07:17.940437 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:17.940488 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20250109 21:07:17.941179 26271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:17.940963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:46765 (local address 127.0.0.1:38200)
0109 21:07:17.941074 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:17.941116 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20250109 21:07:17.941287 26268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.124:46765: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
W20250109 21:07:17.995477 26275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:17.995718 26275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:17.995802 26275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:17.998878 26275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:07:17.998940 26275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:17.998975 26275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:17.999004 26275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:07:17.999028 26275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:07:18.003185 26275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:41127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:40053,127.12.126.125:36789,127.12.126.124:46765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.124.keytab
--principal=kudu/127.12.126.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.124:46765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.126.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:18.004225 26275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:18.005030 26275 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250109 21:07:18.009425 26268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } attempt: 2
I20250109 21:07:18.009721 26193 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } attempt: 2
I20250109 21:07:18.010231 26271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.009921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:46765 (local address 127.0.0.1:38204)
0109 21:07:18.010074 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:18.010138 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20250109 21:07:18.010363 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.010096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:46765 (local address 127.0.0.1:38220)
0109 21:07:18.010223 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:18.010275 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20250109 21:07:18.011109 26271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.010860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:46765 (local address 127.0.0.1:38222)
0109 21:07:18.011014 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:18.011055 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20250109 21:07:18.011262 26268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.124:46765: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
I20250109 21:07:18.011380 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.010957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:46765 (local address 127.0.0.1:38230)
0109 21:07:18.011224 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:18.011317 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
W20250109 21:07:18.011497 26193 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.124:46765: Network error: Client connection negotiation failed: client connection to 127.12.126.124:46765: connect: Connection refused (error 111)
W20250109 21:07:18.012014 26281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:18.012748 26283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:18.013399 26280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:18.014168 26275 server_base.cc:1034] running on GCE node
I20250109 21:07:18.014609 26275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:18.015468 26275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:18.016610 26275 hybrid_clock.cc:648] HybridClock initialized: now 1736456838016599 us; error 21 us; skew 500 ppm
Jan 09 21:07:18 dist-test-slave-g0zb krb5kdc[26112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456838, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:18.019085 26275 init.cc:377] Logged in from keytab as kudu/127.12.126.124@KRBTEST.COM (short username kudu)
I20250109 21:07:18.019917 26275 webserver.cc:458] Webserver started at http://127.12.126.124:42437/ using document root <none> and password file <none>
I20250109 21:07:18.020418 26275 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:18.020468 26275 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:18.020637 26275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:07:18.022352 26275 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-2/data/instance:
uuid: "61eae1765b044fed9bd05ec1c8495dbc"
format_stamp: "Formatted at 2025-01-09 21:07:18 on dist-test-slave-g0zb"
server_key: "f95fb6508ecc81398e836a94d4f83a6d"
server_key_iv: "a49bfd2c4872d988be590a2ddb2f29e9"
server_key_version: "encryptionkey@0"
I20250109 21:07:18.022828 26275 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-2/wal/instance:
uuid: "61eae1765b044fed9bd05ec1c8495dbc"
format_stamp: "Formatted at 2025-01-09 21:07:18 on dist-test-slave-g0zb"
server_key: "f95fb6508ecc81398e836a94d4f83a6d"
server_key_iv: "a49bfd2c4872d988be590a2ddb2f29e9"
server_key_version: "encryptionkey@0"
I20250109 21:07:18.026037 26275 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20250109 21:07:18.028023 26290 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:18.029003 26275 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:07:18.029101 26275 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-2/wal
uuid: "61eae1765b044fed9bd05ec1c8495dbc"
format_stamp: "Formatted at 2025-01-09 21:07:18 on dist-test-slave-g0zb"
server_key: "f95fb6508ecc81398e836a94d4f83a6d"
server_key_iv: "a49bfd2c4872d988be590a2ddb2f29e9"
server_key_version: "encryptionkey@0"
I20250109 21:07:18.029181 26275 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:18.061362 26275 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:18.064534 26275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:18.064704 26275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:18.071130 26275 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.124:46765
I20250109 21:07:18.071147 26342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.124:46765 every 8 connection(s)
I20250109 21:07:18.072093 26275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-2/data/info.pb
I20250109 21:07:18.074753 26343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:18.077208 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 26275
I20250109 21:07:18.077347 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1736456573526088-12793-0/minicluster-data/master-2/wal/instance
I20250109 21:07:18.077605 12793 external_mini_cluster.cc:1453] Setting key d3759c7aa4e6ab13a4a940befed21047
I20250109 21:07:18.078261 26343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } has no permanent_uuid. Determining permanent_uuid...
Jan 09 21:07:18 dist-test-slave-g0zb krb5kdc[26112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456838, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.124@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
Jan 09 21:07:18 dist-test-slave-g0zb krb5kdc[26112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.124@KRBTEST.COM
I20250109 21:07:18.088508 26197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.079967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.12.126.126:40053)
0109 21:07:18.080089 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:18.080093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:18.080356 (+ 263us) server_negotiation.cc:407] Connection header received
0109 21:07:18.080605 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:18.080608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:18.080666 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:18.080741 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:18.081595 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.082072 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:18.082641 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.082834 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:18.085019 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:18.085034 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:18.085036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:18.085060 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:18.086689 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.087212 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.087214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.087216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.087263 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.087550 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.087556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.087559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.087720 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:18.087815 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:18.088236 (+ 421us) server_negotiation.cc:299] Negotiation successful
0109 21:07:18.088350 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250109 21:07:18.089097 26346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.078697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.12.126.124:46765)
0109 21:07:18.079167 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:18.079178 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:18.079216 (+ 38us) server_negotiation.cc:407] Connection header received
0109 21:07:18.079910 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:18.079935 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:18.080453 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:18.080705 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:18.081595 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.082316 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:18.082936 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.083165 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:18.085440 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:18.085456 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:18.085464 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:18.085483 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:18.087061 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.087449 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.087453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.087454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.087495 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.087694 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.087698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.087699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.088034 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:18.088189 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:18.088416 (+ 227us) server_negotiation.cc:299] Negotiation successful
0109 21:07:18.088767 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"spinlock_wait_cycles":28544,"thread_start_us":139,"threads_started":1}
I20250109 21:07:18.089099 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.079875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:40053 (local address 127.0.0.1:46698)
0109 21:07:18.080216 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:18.080246 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:18.080439 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:18.080793 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:18.080798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:18.081167 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:18.081453 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:18.081465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.082185 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:18.082188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:18.082537 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:18.082544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.082736 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:18.083378 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:18.083414 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:18.084892 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:18.086792 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:18.086823 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:18.086831 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:18.087109 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:18.087344 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:18.087347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:18.087349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:18.087465 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:18.087816 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:18.087821 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:18.088007 (+ 186us) client_negotiation.cc:769] Sending connection context
0109 21:07:18.088167 (+ 160us) client_negotiation.cc:240] Negotiation successful
0109 21:07:18.088418 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":115,"threads_started":1}
I20250109 21:07:18.090183 26343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } has no permanent_uuid. Determining permanent_uuid...
Jan 09 21:07:18 dist-test-slave-g0zb krb5kdc[26112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456838, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.124@KRBTEST.COM for kudu/127.12.126.125@KRBTEST.COM
Jan 09 21:07:18 dist-test-slave-g0zb krb5kdc[26112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:07:18.098573 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.090618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:36789 (local address 127.0.0.1:36012)
0109 21:07:18.090718 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:18.090731 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:18.090822 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:18.091346 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:18.091348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:18.091519 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:18.091742 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:18.091749 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.092588 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:18.092590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:18.093040 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:18.093049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.093146 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:18.093848 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:18.093866 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:18.095149 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:18.097102 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:18.097105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:18.097106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:18.097292 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:18.097537 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:18.097540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:18.097541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:18.097609 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:18.097968 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:18.097971 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:18.098057 (+ 86us) client_negotiation.cc:769] Sending connection context
0109 21:07:18.098270 (+ 213us) client_negotiation.cc:240] Negotiation successful
0109 21:07:18.098390 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20250109 21:07:18.098615 26273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.090714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36012 (local address 127.12.126.125:36789)
0109 21:07:18.090864 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:18.090869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:18.090883 (+ 14us) server_negotiation.cc:407] Connection header received
0109 21:07:18.091092 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:18.091098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:18.091159 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:18.091207 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:18.091868 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.092491 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:18.093151 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.093327 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:18.095293 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:18.095316 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:18.095321 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:18.095351 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:18.096923 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.097390 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.097393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.097394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.097446 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.097702 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.097705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.097706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.097873 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:18.097974 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:18.098352 (+ 378us) server_negotiation.cc:299] Negotiation successful
0109 21:07:18.098479 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250109 21:07:18.099347 26197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.090916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46712 (local address 127.12.126.126:40053)
0109 21:07:18.091026 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:18.091029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:18.091040 (+ 11us) server_negotiation.cc:407] Connection header received
0109 21:07:18.091140 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:18.091142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:18.091172 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:18.091238 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:18.092028 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.092610 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:18.093232 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.093377 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:18.095820 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:18.095839 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:18.095843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:18.095873 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:18.097419 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.097832 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.097836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.097838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.097880 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.098117 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.098120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.098121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.098305 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:18.098503 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:18.099094 (+ 591us) server_negotiation.cc:299] Negotiation successful
0109 21:07:18.099211 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250109 21:07:18.099371 26343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } has no permanent_uuid. Determining permanent_uuid...
I20250109 21:07:18.102429 26268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } attempt: 3
Jan 09 21:07:18 dist-test-slave-g0zb krb5kdc[26112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456838, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.124@KRBTEST.COM for kudu/127.12.126.124@KRBTEST.COM
Jan 09 21:07:18 dist-test-slave-g0zb krb5kdc[26112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.125@KRBTEST.COM
I20250109 21:07:18.106987 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.099798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:46765 (local address 127.0.0.1:38246)
0109 21:07:18.099902 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:18.099914 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:18.099988 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:18.100243 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:18.100246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:18.100395 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:18.100581 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:18.100589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.101492 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:18.101497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:18.101984 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:18.101992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.102102 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:18.102676 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:18.102696 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:18.104053 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:18.105598 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:18.105601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:18.105603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:18.105753 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:18.105970 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:18.105973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:18.105975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:18.106031 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:18.106443 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:18.106446 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:18.106525 (+ 79us) client_negotiation.cc:769] Sending connection context
0109 21:07:18.106637 (+ 112us) client_negotiation.cc:240] Negotiation successful
0109 21:07:18.106756 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20250109 21:07:18.107409 26346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.099924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.12.126.124:46765)
0109 21:07:18.100058 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:18.100062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:18.100074 (+ 12us) server_negotiation.cc:407] Connection header received
0109 21:07:18.100107 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:18.100110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:18.100146 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:18.100213 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:18.100699 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.101383 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:18.102109 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.102604 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:18.104160 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:18.104172 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:18.104174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:18.104192 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:18.105510 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.105842 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.105844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.105846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.105883 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.106133 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.106138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.106141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.106344 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:18.106446 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:18.107215 (+ 769us) server_negotiation.cc:299] Negotiation successful
0109 21:07:18.107321 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Jan 09 21:07:18 dist-test-slave-g0zb krb5kdc[26112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456837, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.125@KRBTEST.COM for kudu/127.12.126.124@KRBTEST.COM
I20250109 21:07:18.109102 26273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.100529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.12.126.125:36789)
0109 21:07:18.100659 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:18.100664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:18.100678 (+ 14us) server_negotiation.cc:407] Connection header received
0109 21:07:18.100721 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:18.100724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:18.100761 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:18.100814 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:18.101534 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.102141 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:18.102982 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.103186 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:18.105839 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:18.105853 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:18.105856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:18.105878 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:18.107673 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.108074 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.108078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.108080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.108124 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.108356 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.108359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.108361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.108486 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:18.108633 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:18.108818 (+ 185us) server_negotiation.cc:299] Negotiation successful
0109 21:07:18.108932 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250109 21:07:18.110638 26193 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } attempt: 3
I20250109 21:07:18.110898 26343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc: Bootstrap starting.
I20250109 21:07:18.111927 26271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.102787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:46765 (local address 127.0.0.1:38254)
0109 21:07:18.102966 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:18.102979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:18.103067 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:18.103973 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:18.103976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:18.104161 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:18.104397 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:18.104407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.105837 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:18.105842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:18.106301 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:18.106313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.106393 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:18.106864 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:18.106880 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:18.108380 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:18.110465 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:18.110468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:18.110470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:18.110675 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:18.111047 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:18.111050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:18.111053 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:18.111109 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:18.111513 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:18.111516 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:18.111598 (+ 82us) client_negotiation.cc:769] Sending connection context
0109 21:07:18.111679 (+ 81us) client_negotiation.cc:240] Negotiation successful
0109 21:07:18.111795 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20250109 21:07:18.112322 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:07:18.112715 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.103477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38254 (local address 127.12.126.124:46765)
0109 21:07:18.103743 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:18.103747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:18.103763 (+ 16us) server_negotiation.cc:407] Connection header received
0109 21:07:18.103796 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:18.103801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:18.103842 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:18.103934 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:18.104548 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.105706 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:18.106610 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.106840 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:18.108501 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:18.108515 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:18.108518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:18.108540 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:18.110348 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.110830 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.110833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.110835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.110884 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.111220 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.111223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.111225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.111402 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:18.112344 (+ 942us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:18.112466 (+ 122us) server_negotiation.cc:299] Negotiation successful
0109 21:07:18.112565 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":102,"threads_started":1}
I20250109 21:07:18.113520 26343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc: Neither blocks nor log segments found. Creating new log.
I20250109 21:07:18.114403 26343 log.cc:826] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc: Log is configured to *not* fsync() on all Append() calls
Jan 09 21:07:18 dist-test-slave-g0zb krb5kdc[26112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456837, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for kudu/127.12.126.124@KRBTEST.COM
I20250109 21:07:18.117334 26268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058: Bootstrap starting.
I20250109 21:07:18.117362 26343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc: No bootstrap required, opened a new log
I20250109 21:07:18.119694 26268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058: Neither blocks nor log segments found. Creating new log.
I20250109 21:07:18.120555 26268 log.cc:826] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:18.121484 26343 raft_consensus.cc:357] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } }
I20250109 21:07:18.121706 26343 raft_consensus.cc:383] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:18.121790 26343 raft_consensus.cc:738] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61eae1765b044fed9bd05ec1c8495dbc, State: Initialized, Role: FOLLOWER
I20250109 21:07:18.122402 26343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } }
I20250109 21:07:18.122967 26268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058: No bootstrap required, opened a new log
I20250109 21:07:18.124442 26273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.115128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36026 (local address 127.12.126.125:36789)
0109 21:07:18.115232 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:18.115236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:18.115614 (+ 378us) server_negotiation.cc:407] Connection header received
0109 21:07:18.115783 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:18.115787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:18.115839 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:18.115917 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:18.116826 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.117414 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:18.118175 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.118364 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:18.120579 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:18.120594 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:18.120598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:18.120623 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:18.122758 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.123226 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.123230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.123232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.123278 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.123544 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.123547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.123549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.123722 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:18.123825 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:18.124153 (+ 328us) server_negotiation.cc:299] Negotiation successful
0109 21:07:18.124264 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250109 21:07:18.125008 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.111010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:46765 (local address 127.0.0.1:38256)
0109 21:07:18.111227 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:18.111240 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:18.111321 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:18.111717 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:18.111720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:18.111889 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:18.112115 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:18.112124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.113144 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:18.113148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:18.113647 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:18.113658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.113753 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:18.114230 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:18.114248 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:18.115542 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:18.123534 (+ 7992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:18.123537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:18.123539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:18.123757 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:18.124038 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:18.124056 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:18.124059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:18.124119 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:18.124521 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:18.124525 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:18.124624 (+ 99us) client_negotiation.cc:769] Sending connection context
0109 21:07:18.124710 (+ 86us) client_negotiation.cc:240] Negotiation successful
0109 21:07:18.124840 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20250109 21:07:18.125443 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.114493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.12.126.124:46765)
0109 21:07:18.116274 (+ 1781us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:18.116277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:18.116289 (+ 12us) server_negotiation.cc:407] Connection header received
0109 21:07:18.116319 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:18.116321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:18.116349 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:18.116414 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:18.117364 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.117944 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:18.118745 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.118944 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:18.120662 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:18.120677 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:18.120680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:18.120705 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:18.122598 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.123864 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.123867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.123869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.123916 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.124168 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.124171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.124173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.124336 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:18.124457 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:18.124974 (+ 517us) server_negotiation.cc:299] Negotiation successful
0109 21:07:18.125304 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1709}
I20250109 21:07:18.125633 26346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.111435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.12.126.124:46765)
0109 21:07:18.111528 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:18.111531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:18.111542 (+ 11us) server_negotiation.cc:407] Connection header received
0109 21:07:18.111570 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:18.111573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:18.111606 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:18.112231 (+ 625us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:18.112312 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.112868 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:18.114045 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.115406 (+ 1361us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:18.115649 (+ 243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:18.115662 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:18.115665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:18.115686 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:18.123435 (+ 7749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.123868 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.123872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.123874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.123933 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.124230 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.124235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.124237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.124405 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:18.125300 (+ 895us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:18.125375 (+ 75us) server_negotiation.cc:299] Negotiation successful
0109 21:07:18.125475 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38,"spinlock_wait_cycles":1280}
I20250109 21:07:18.125928 26343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:07:18.127574 26358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } } }
I20250109 21:07:18.127718 26358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:07:18.128459 26193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2: Bootstrap starting.
I20250109 21:07:18.132017 26193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2: Neither blocks nor log segments found. Creating new log.
W20250109 21:07:18.132239 26369 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:07:18.132310 26369 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 21:07:18.130033 26275 master_runner.cc:186] Error getting master registration for 127.12.126.126:40053: OK, instance_id { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" instance_seqno: 1736456837652609 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250109 21:07:18.132469 26275 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 21:07:18.132872 26193 log.cc:826] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:18.133824 26197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:18.121228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46716 (local address 127.12.126.126:40053)
0109 21:07:18.121349 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:18.121353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:18.126090 (+ 4737us) server_negotiation.cc:407] Connection header received
0109 21:07:18.126254 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:18.126257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:18.126320 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:18.126371 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:18.127435 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.128090 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:18.128879 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:18.129103 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:18.130351 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:18.130367 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:18.130370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:18.130395 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:18.132232 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.132735 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.132738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.132739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.132781 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:18.133033 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:18.133037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:18.133038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:18.133210 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:18.133335 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:18.133575 (+ 240us) server_negotiation.cc:299] Negotiation successful
0109 21:07:18.133683 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250109 21:07:18.133975 26268 raft_consensus.cc:357] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } }
I20250109 21:07:18.134171 26268 raft_consensus.cc:383] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:18.134238 26268 raft_consensus.cc:738] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d23ea953dd4903ba7dd47a8220c058, State: Initialized, Role: FOLLOWER
I20250109 21:07:18.134670 26268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } }
I20250109 21:07:18.135085 26371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } } }
I20250109 21:07:18.135196 26371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:07:18.135862 26268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:07:18.136798 26193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2: No bootstrap required, opened a new log
I20250109 21:07:18.140091 26193 raft_consensus.cc:357] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } }
I20250109 21:07:18.140233 26193 raft_consensus.cc:383] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:18.140268 26193 raft_consensus.cc:738] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7ebf6b2be1e46b087668033126a9de2, State: Initialized, Role: FOLLOWER
I20250109 21:07:18.140627 26193 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } }
I20250109 21:07:18.141016 26384 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } } }
I20250109 21:07:18.141113 26384 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:07:18.141589 26193 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:07:18.140446 26200 master_runner.cc:186] Error getting master registration for 127.12.126.126:40053: OK, instance_id { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" instance_seqno: 1736456837652609 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20250109 21:07:18.146061 26383 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:07:18.146060 26395 catalog_manager.cc:1559] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:07:18.146123 26383 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250109 21:07:18.146123 26395 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 21:07:18.148958 26200 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 21:07:18.149247 26146 tablet.cc:2367] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:07:18.150734 26125 master_runner.cc:263] No leader master found from master c7ebf6b2be1e46b087668033126a9de2
I20250109 21:07:18.150821 26125 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 21:07:18.315886 26296 tablet.cc:2367] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250109 21:07:18.352985 26221 tablet.cc:2367] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:07:18.453207 26371 raft_consensus.cc:491] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:07:18.453399 26371 raft_consensus.cc:513] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } }
I20250109 21:07:18.454267 26371 leader_election.cc:290] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7ebf6b2be1e46b087668033126a9de2 (127.12.126.126:40053), 61eae1765b044fed9bd05ec1c8495dbc (127.12.126.124:46765)
I20250109 21:07:18.454880 26167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31d23ea953dd4903ba7dd47a8220c058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7ebf6b2be1e46b087668033126a9de2" is_pre_election: true
I20250109 21:07:18.454880 26317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31d23ea953dd4903ba7dd47a8220c058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61eae1765b044fed9bd05ec1c8495dbc" is_pre_election: true
I20250109 21:07:18.455211 26317 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31d23ea953dd4903ba7dd47a8220c058 in term 0.
I20250109 21:07:18.455211 26167 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31d23ea953dd4903ba7dd47a8220c058 in term 0.
I20250109 21:07:18.455646 26218 leader_election.cc:304] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d23ea953dd4903ba7dd47a8220c058, 61eae1765b044fed9bd05ec1c8495dbc; no voters:
I20250109 21:07:18.455885 26371 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:07:18.455962 26371 raft_consensus.cc:491] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:07:18.456036 26371 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:18.457157 26371 raft_consensus.cc:513] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } }
I20250109 21:07:18.457563 26371 leader_election.cc:290] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [CANDIDATE]: Term 1 election: Requested vote from peers c7ebf6b2be1e46b087668033126a9de2 (127.12.126.126:40053), 61eae1765b044fed9bd05ec1c8495dbc (127.12.126.124:46765)
I20250109 21:07:18.457859 26317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31d23ea953dd4903ba7dd47a8220c058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61eae1765b044fed9bd05ec1c8495dbc"
I20250109 21:07:18.458004 26317 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:18.457988 26167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31d23ea953dd4903ba7dd47a8220c058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7ebf6b2be1e46b087668033126a9de2"
I20250109 21:07:18.458098 26167 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:18.458971 26317 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d23ea953dd4903ba7dd47a8220c058 in term 1.
I20250109 21:07:18.459148 26167 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d23ea953dd4903ba7dd47a8220c058 in term 1.
I20250109 21:07:18.459275 26218 leader_election.cc:304] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d23ea953dd4903ba7dd47a8220c058, 61eae1765b044fed9bd05ec1c8495dbc; no voters:
I20250109 21:07:18.459478 26371 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:07:18.459656 26371 raft_consensus.cc:695] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [term 1 LEADER]: Becoming Leader. State: Replica: 31d23ea953dd4903ba7dd47a8220c058, State: Running, Role: LEADER
I20250109 21:07:18.459944 26371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } }
I20250109 21:07:18.461586 26398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31d23ea953dd4903ba7dd47a8220c058. Latest consensus state: current_term: 1 leader_uuid: "31d23ea953dd4903ba7dd47a8220c058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } } }
I20250109 21:07:18.461685 26398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [sys.catalog]: This master's current role is: LEADER
I20250109 21:07:18.462025 26400 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:07:18.462854 26400 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:07:18.466684 26167 raft_consensus.cc:1270] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2 [term 1 FOLLOWER]: Refusing update from remote peer 31d23ea953dd4903ba7dd47a8220c058: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:07:18.466708 26317 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc [term 1 FOLLOWER]: Refusing update from remote peer 31d23ea953dd4903ba7dd47a8220c058: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:07:18.467259 26371 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:07:18.467382 26398 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:07:18.470518 26384 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31d23ea953dd4903ba7dd47a8220c058. Latest consensus state: current_term: 1 leader_uuid: "31d23ea953dd4903ba7dd47a8220c058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } } }
I20250109 21:07:18.470607 26384 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:07:18.470635 26358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31d23ea953dd4903ba7dd47a8220c058. Latest consensus state: current_term: 1 leader_uuid: "31d23ea953dd4903ba7dd47a8220c058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } } }
I20250109 21:07:18.470746 26358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:07:18.472676 26398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "31d23ea953dd4903ba7dd47a8220c058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } } }
I20250109 21:07:18.472774 26398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [sys.catalog]: This master's current role is: LEADER
I20250109 21:07:18.473059 26398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "31d23ea953dd4903ba7dd47a8220c058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } } }
I20250109 21:07:18.473150 26398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058 [sys.catalog]: This master's current role is: LEADER
I20250109 21:07:18.473145 26403 mvcc.cc:204] Tried to move back new op lower bound from 7112527210354606080 to 7112527210334081024. Current Snapshot: MvccSnapshot[applied={T|T < 7112527210354606080}]
I20250109 21:07:18.473291 26384 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "31d23ea953dd4903ba7dd47a8220c058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } } }
I20250109 21:07:18.473376 26384 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:07:18.473980 26400 catalog_manager.cc:1348] Generated new cluster ID: 450d39f2207a498284be9bd5c7799063
I20250109 21:07:18.474056 26400 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:07:18.474125 26358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "31d23ea953dd4903ba7dd47a8220c058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ebf6b2be1e46b087668033126a9de2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 40053 } } peers { permanent_uuid: "31d23ea953dd4903ba7dd47a8220c058" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 36789 } } peers { permanent_uuid: "61eae1765b044fed9bd05ec1c8495dbc" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 46765 } } }
I20250109 21:07:18.474218 26358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:07:18.474315 26402 mvcc.cc:204] Tried to move back new op lower bound from 7112527210354606080 to 7112527210334081024. Current Snapshot: MvccSnapshot[applied={T|T < 7112527210354606080}]
I20250109 21:07:18.482672 26400 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:07:18.483772 26400 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:07:18.491638 26400 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 31d23ea953dd4903ba7dd47a8220c058: Generated new TSK 0
I20250109 21:07:18.492450 26400 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:07:19.133641 26369 catalog_manager.cc:1260] Loaded cluster ID: 450d39f2207a498284be9bd5c7799063
I20250109 21:07:19.133728 26369 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc: loading cluster ID for follower catalog manager: success
I20250109 21:07:19.135452 26369 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc: acquiring CA information for follower catalog manager: success
I20250109 21:07:19.136612 26369 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 61eae1765b044fed9bd05ec1c8495dbc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 21:07:19.147301 26395 catalog_manager.cc:1260] Loaded cluster ID: 450d39f2207a498284be9bd5c7799063
I20250109 21:07:19.147370 26395 catalog_manager.cc:1553] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2: loading cluster ID for follower catalog manager: success
I20250109 21:07:19.148972 26395 catalog_manager.cc:1575] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2: acquiring CA information for follower catalog manager: success
I20250109 21:07:19.150058 26395 catalog_manager.cc:1603] T 00000000000000000000000000000000 P c7ebf6b2be1e46b087668033126a9de2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 21:07:19.559662 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 26125
I20250109 21:07:19.566171 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 26200
I20250109 21:07:19.572127 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 26275
2025-01-09T21:07:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3228 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:07:19 dist-test-slave-g0zb krb5kdc[26416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:07:19 dist-test-slave-g0zb krb5kdc[26416](info): set up 2 sockets
Jan 09 21:07:19 dist-test-slave-g0zb krb5kdc[26416](info): commencing operation
krb5kdc: starting...
W20250109 21:07:20.579421 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:07:20 dist-test-slave-g0zb krb5kdc[26416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:07:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:07:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:07:20.687525 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:37811
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:37811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:07:20.788545 26429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:20.788784 26429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:20.788857 26429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:20.791764 26429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:07:20.791815 26429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:20.791847 26429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:20.791875 26429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:07:20.791911 26429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:07:20.795725 26429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:37811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:37811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:20.796643 26429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:20.797391 26429 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:20.803954 26437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:20.804611 26434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:20.805195 26435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:20.806236 26429 server_base.cc:1034] running on GCE node
I20250109 21:07:20.806630 26429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:20.807425 26429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:20.808555 26429 hybrid_clock.cc:648] HybridClock initialized: now 1736456840808540 us; error 33 us; skew 500 ppm
Jan 09 21:07:20 dist-test-slave-g0zb krb5kdc[26416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456840, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:20.811049 26429 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:07:20.811918 26429 webserver.cc:458] Webserver started at http://127.12.126.126:40405/ using document root <none> and password file <none>
I20250109 21:07:20.812423 26429 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:20.812474 26429 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:20.812644 26429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:07:20.814285 26429 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "fb0d4a5121304a5fa7872d8de3596ab8"
format_stamp: "Formatted at 2025-01-09 21:07:20 on dist-test-slave-g0zb"
server_key: "d110bf35ea6d7dcad1d326e09e5aac3b"
server_key_iv: "2920cf85a64ea9d1ea0f4acddb4062a2"
server_key_version: "encryptionkey@0"
I20250109 21:07:20.814736 26429 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "fb0d4a5121304a5fa7872d8de3596ab8"
format_stamp: "Formatted at 2025-01-09 21:07:20 on dist-test-slave-g0zb"
server_key: "d110bf35ea6d7dcad1d326e09e5aac3b"
server_key_iv: "2920cf85a64ea9d1ea0f4acddb4062a2"
server_key_version: "encryptionkey@0"
I20250109 21:07:20.817957 26429 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250109 21:07:20.819976 26444 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:20.820896 26429 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:07:20.820997 26429 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "fb0d4a5121304a5fa7872d8de3596ab8"
format_stamp: "Formatted at 2025-01-09 21:07:20 on dist-test-slave-g0zb"
server_key: "d110bf35ea6d7dcad1d326e09e5aac3b"
server_key_iv: "2920cf85a64ea9d1ea0f4acddb4062a2"
server_key_version: "encryptionkey@0"
I20250109 21:07:20.821089 26429 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:20.841742 26429 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:20.844317 26429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:20.844468 26429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:20.850970 26429 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:37811
I20250109 21:07:20.851190 26496 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:37811 every 8 connection(s)
I20250109 21:07:20.851959 26429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:07:20.854333 26497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:20.859287 26497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8: Bootstrap starting.
I20250109 21:07:20.861400 26497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8: Neither blocks nor log segments found. Creating new log.
I20250109 21:07:20.862071 26497 log.cc:826] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:20.862149 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 26429
I20250109 21:07:20.862280 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:07:20.862533 12793 external_mini_cluster.cc:1453] Setting key fb3a951fc04757e0fbf90ccab4708611
I20250109 21:07:20.863981 26497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8: No bootstrap required, opened a new log
I20250109 21:07:20.866796 26497 raft_consensus.cc:357] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0d4a5121304a5fa7872d8de3596ab8" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 37811 } }
I20250109 21:07:20.867003 26497 raft_consensus.cc:383] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:20.867076 26497 raft_consensus.cc:738] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0d4a5121304a5fa7872d8de3596ab8, State: Initialized, Role: FOLLOWER
I20250109 21:07:20.867475 26497 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0d4a5121304a5fa7872d8de3596ab8" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 37811 } }
I20250109 21:07:20.867594 26497 raft_consensus.cc:397] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:07:20.867655 26497 raft_consensus.cc:491] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:07:20.867758 26497 raft_consensus.cc:3054] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:20.868641 26497 raft_consensus.cc:513] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0d4a5121304a5fa7872d8de3596ab8" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 37811 } }
I20250109 21:07:20.868944 26497 leader_election.cc:304] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb0d4a5121304a5fa7872d8de3596ab8; no voters:
I20250109 21:07:20.869210 26497 leader_election.cc:290] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:07:20.869333 26502 raft_consensus.cc:2798] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:07:20.869546 26502 raft_consensus.cc:695] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8 [term 1 LEADER]: Becoming Leader. State: Replica: fb0d4a5121304a5fa7872d8de3596ab8, State: Running, Role: LEADER
I20250109 21:07:20.869922 26502 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0d4a5121304a5fa7872d8de3596ab8" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 37811 } }
Jan 09 21:07:20 dist-test-slave-g0zb krb5kdc[26416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:07:20.870215 26497 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:07:20.871105 26504 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb0d4a5121304a5fa7872d8de3596ab8. Latest consensus state: current_term: 1 leader_uuid: "fb0d4a5121304a5fa7872d8de3596ab8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0d4a5121304a5fa7872d8de3596ab8" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 37811 } } }
I20250109 21:07:20.871217 26504 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8 [sys.catalog]: This master's current role is: LEADER
I20250109 21:07:20.871861 26503 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb0d4a5121304a5fa7872d8de3596ab8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0d4a5121304a5fa7872d8de3596ab8" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 37811 } } }
I20250109 21:07:20.872001 26503 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8 [sys.catalog]: This master's current role is: LEADER
I20250109 21:07:20.872733 26511 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:07:20.875444 26511 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:07:20.875862 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:20.863496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.12.126.126:37811)
0109 21:07:20.863838 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:20.863846 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:20.863878 (+ 32us) server_negotiation.cc:407] Connection header received
0109 21:07:20.864436 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:20.864454 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:20.864712 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:20.864983 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:20.866029 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:20.867265 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:20.867963 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:20.868257 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:20.870849 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:20.870871 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:20.870886 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:20.870933 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:20.873175 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:20.873627 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:20.873632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:20.873636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:20.873719 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:20.873974 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:20.873977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:20.873979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:20.874369 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:20.874571 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:20.874837 (+ 266us) server_negotiation.cc:299] Negotiation successful
0109 21:07:20.875136 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":93,"threads_started":1}
I20250109 21:07:20.878710 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:07:20.880785 26511 catalog_manager.cc:1348] Generated new cluster ID: b262670aa4a94bc3b1048e2ae6d134bd
I20250109 21:07:20.880847 26511 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:07:20.887781 26511 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:07:20.888597 26511 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:07:20.895787 26511 catalog_manager.cc:5899] T 00000000000000000000000000000000 P fb0d4a5121304a5fa7872d8de3596ab8: Generated new TSK 0
I20250109 21:07:20.896416 26511 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:07:20.936251 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 26429
2025-01-09T21:07:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1363 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:07:20 dist-test-slave-g0zb krb5kdc[26524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:07:20 dist-test-slave-g0zb krb5kdc[26524](info): set up 2 sockets
Jan 09 21:07:20 dist-test-slave-g0zb krb5kdc[26524](info): commencing operation
krb5kdc: starting...
W20250109 21:07:21.955540 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:07:22 dist-test-slave-g0zb krb5kdc[26524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:07:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:07:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:07:22.070189 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:33113
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:33113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:07:22.173775 26537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:22.174038 26537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:22.174125 26537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:22.177254 26537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:07:22.177315 26537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:22.177353 26537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:22.177393 26537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:07:22.177431 26537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:07:22.181586 26537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:33113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:33113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:22.182619 26537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:22.183446 26537 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:22.190052 26545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:22.190814 26543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:22.191368 26542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:22.192126 26537 server_base.cc:1034] running on GCE node
I20250109 21:07:22.192553 26537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:22.193390 26537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:22.194548 26537 hybrid_clock.cc:648] HybridClock initialized: now 1736456842194522 us; error 37 us; skew 500 ppm
Jan 09 21:07:22 dist-test-slave-g0zb krb5kdc[26524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456842, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:22.202392 26537 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:07:22.203274 26537 webserver.cc:458] Webserver started at http://127.12.126.126:46511/ using document root <none> and password file <none>
I20250109 21:07:22.203774 26537 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:22.203825 26537 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:22.203971 26537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:07:22.205699 26537 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "72863e5fa41f5e8b9c842c8f307bad38"
server_key_iv: "f06545f6132740a1c757654e97bd4ae3"
server_key_version: "encryptionkey@0"
I20250109 21:07:22.206136 26537 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "72863e5fa41f5e8b9c842c8f307bad38"
server_key_iv: "f06545f6132740a1c757654e97bd4ae3"
server_key_version: "encryptionkey@0"
I20250109 21:07:22.209329 26537 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20250109 21:07:22.211334 26552 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:22.212374 26537 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:07:22.212487 26537 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "72863e5fa41f5e8b9c842c8f307bad38"
server_key_iv: "f06545f6132740a1c757654e97bd4ae3"
server_key_version: "encryptionkey@0"
I20250109 21:07:22.212579 26537 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:22.222898 26537 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:22.225939 26537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:22.226089 26537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:22.231962 26537 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:33113
I20250109 21:07:22.232209 26604 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:33113 every 8 connection(s)
I20250109 21:07:22.232903 26537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:07:22.235345 26605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:22.235414 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 26537
I20250109 21:07:22.235533 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:07:22.235809 12793 external_mini_cluster.cc:1453] Setting key 58ac14758e3574a1b6ae06a51a518712
I20250109 21:07:22.241588 26605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f: Bootstrap starting.
Jan 09 21:07:22 dist-test-slave-g0zb krb5kdc[26524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:07:22.243885 26605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f: Neither blocks nor log segments found. Creating new log.
I20250109 21:07:22.244617 26605 log.cc:826] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:22.246590 26605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f: No bootstrap required, opened a new log
I20250109 21:07:22.248766 26608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:22.236929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35888 (local address 127.12.126.126:33113)
0109 21:07:22.237316 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:22.237329 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:22.237368 (+ 39us) server_negotiation.cc:407] Connection header received
0109 21:07:22.238280 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:22.238304 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:22.238651 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:22.239033 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:22.239799 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.241015 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:22.241607 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.241870 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:22.244150 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:22.244166 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:22.244175 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:22.244206 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:22.246217 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:22.246599 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:22.246608 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:22.246615 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:22.246704 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:22.246963 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:22.246968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:22.246971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:22.247432 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:22.247613 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:22.247800 (+ 187us) server_negotiation.cc:299] Negotiation successful
0109 21:07:22.248054 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":113,"threads_started":1}
I20250109 21:07:22.249907 26605 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33113 } }
I20250109 21:07:22.250119 26605 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:22.250180 26605 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0e9f0299b74ce2ba7c731714c2cf6f, State: Initialized, Role: FOLLOWER
I20250109 21:07:22.250591 26605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33113 } }
I20250109 21:07:22.250717 26605 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:07:22.250782 26605 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:07:22.250912 26605 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:22.251641 26605 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33113 } }
I20250109 21:07:22.251933 26605 leader_election.cc:304] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f0e9f0299b74ce2ba7c731714c2cf6f; no voters:
I20250109 21:07:22.252193 26605 leader_election.cc:290] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:07:22.252445 26610 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:07:22.252642 26610 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 1 LEADER]: Becoming Leader. State: Replica: 8f0e9f0299b74ce2ba7c731714c2cf6f, State: Running, Role: LEADER
I20250109 21:07:22.252904 26610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33113 } }
I20250109 21:07:22.253291 26605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:07:22.254006 26610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f0e9f0299b74ce2ba7c731714c2cf6f. Latest consensus state: current_term: 1 leader_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33113 } } }
I20250109 21:07:22.254190 26610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [sys.catalog]: This master's current role is: LEADER
I20250109 21:07:22.254374 26612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33113 } } }
I20250109 21:07:22.254513 26612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [sys.catalog]: This master's current role is: LEADER
I20250109 21:07:22.254787 26619 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:07:22.257530 26619 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:07:22.262171 26619 catalog_manager.cc:1348] Generated new cluster ID: ee0104f07acb45af8684c5ff9af5d1c7
I20250109 21:07:22.262233 26619 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:07:22.271088 26619 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:07:22.272205 26619 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:07:22.286675 26619 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f: Generated new TSK 0
I20250109 21:07:22.287422 26619 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
I20250109 21:07:22.315431 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:33113
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:07:22.415391 26631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:22.415589 26631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:22.415659 26631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:22.418605 26631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:22.418658 26631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:22.418726 26631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:07:22.422649 26631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:22.423565 26631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:22.424337 26631 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:22.431840 26637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:22.432364 26636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:22.432902 26639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:22.433934 26631 server_base.cc:1034] running on GCE node
I20250109 21:07:22.434229 26631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:22.434675 26631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:22.435803 26631 hybrid_clock.cc:648] HybridClock initialized: now 1736456842435789 us; error 35 us; skew 500 ppm
Jan 09 21:07:22 dist-test-slave-g0zb krb5kdc[26524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456842, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:22.438134 26631 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:07:22.438906 26631 webserver.cc:458] Webserver started at http://127.12.126.65:35675/ using document root <none> and password file <none>
I20250109 21:07:22.439429 26631 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:22.439487 26631 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:22.439661 26631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:07:22.441264 26631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "cd59d4d98f404eb6ab752bd586912096"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "e3749b2e75136e80788ab3049b38d2ef"
server_key_iv: "6e7ca46d61ba11a4bfc6aa76fc8d3865"
server_key_version: "encryptionkey@0"
I20250109 21:07:22.441637 26631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "cd59d4d98f404eb6ab752bd586912096"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "e3749b2e75136e80788ab3049b38d2ef"
server_key_iv: "6e7ca46d61ba11a4bfc6aa76fc8d3865"
server_key_version: "encryptionkey@0"
I20250109 21:07:22.444653 26631 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20250109 21:07:22.446594 26646 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:22.447525 26631 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:07:22.447614 26631 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "cd59d4d98f404eb6ab752bd586912096"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "e3749b2e75136e80788ab3049b38d2ef"
server_key_iv: "6e7ca46d61ba11a4bfc6aa76fc8d3865"
server_key_version: "encryptionkey@0"
I20250109 21:07:22.447700 26631 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:22.461907 26631 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:22.464695 26631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:22.464836 26631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:22.465302 26631 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:07:22.465992 26631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:07:22.466041 26631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:22.466092 26631 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:07:22.466120 26631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:22.473997 26631 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:34057
I20250109 21:07:22.474068 26759 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:34057 every 8 connection(s)
I20250109 21:07:22.474864 26631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:07:22.480226 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 26631
I20250109 21:07:22.480374 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:07:22.480638 12793 external_mini_cluster.cc:1453] Setting key c95eb1045f3944aa52a0992eb112f8c5
Jan 09 21:07:22 dist-test-slave-g0zb krb5kdc[26524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456842, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:07:22.487027 26608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:22.476404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:38445 (local address 127.12.126.126:33113)
0109 21:07:22.476533 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:22.476537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:22.477614 (+ 1077us) server_negotiation.cc:407] Connection header received
0109 21:07:22.478399 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:22.478402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:22.478448 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:22.478520 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:22.479678 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.480168 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:22.481000 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.481198 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:22.483491 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:22.483512 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:22.483517 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:22.483549 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:22.485151 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:22.485634 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:22.485641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:22.485642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:22.485694 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:22.485910 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:22.485914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:22.485918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:22.486145 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:22.486249 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:22.486622 (+ 373us) server_negotiation.cc:299] Negotiation successful
0109 21:07:22.486756 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250109 21:07:22.487404 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:22.477019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:38445)
0109 21:07:22.477459 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:22.477500 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:22.478208 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:22.478644 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:22.478651 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:22.479010 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:22.479558 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:22.479569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.480308 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:22.480312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:22.480860 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:22.480871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.481087 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:22.481641 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:22.481664 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:22.483307 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:22.485254 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:22.485271 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:22.485278 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:22.485540 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:22.485773 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:22.485775 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:22.485776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:22.485842 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:22.486231 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:22.486235 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:22.486415 (+ 180us) client_negotiation.cc:769] Sending connection context
0109 21:07:22.486553 (+ 138us) client_negotiation.cc:240] Negotiation successful
0109 21:07:22.486730 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20250109 21:07:22.488418 26760 heartbeater.cc:346] Connected to a master server at 127.12.126.126:33113
I20250109 21:07:22.488607 26760 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:22.489027 26760 heartbeater.cc:510] Master 127.12.126.126:33113 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.126.66@KRBTEST.COM; defaulting to no policy
I20250109 21:07:22.490397 26567 ts_manager.cc:194] Registered new tserver with Master: cd59d4d98f404eb6ab752bd586912096 (127.12.126.65:34057)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.66@KRBTEST.COM" created.
I20250109 21:07:22.491580 26567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:38445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
I20250109 21:07:22.505245 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:33113
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:07:22.605764 26765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:22.605976 26765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:22.606060 26765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:22.609086 26765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:22.609167 26765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:22.609287 26765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:07:22.613240 26765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:22.614173 26765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:22.615005 26765 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:22.623049 26773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:22.623502 26770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:22.623804 26771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:22.624819 26765 server_base.cc:1034] running on GCE node
I20250109 21:07:22.625131 26765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:22.625635 26765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:22.626778 26765 hybrid_clock.cc:648] HybridClock initialized: now 1736456842626756 us; error 34 us; skew 500 ppm
Jan 09 21:07:22 dist-test-slave-g0zb krb5kdc[26524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456842, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:22.629298 26765 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:07:22.630172 26765 webserver.cc:458] Webserver started at http://127.12.126.66:37509/ using document root <none> and password file <none>
I20250109 21:07:22.630703 26765 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:22.630759 26765 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:22.630972 26765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:07:22.632651 26765 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "fd658876396e4ee6a9097fe40fbb39af"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "cdb29647d56ea3a6c64565f7c376763d"
server_key_iv: "f1aa231fa9758b0f7b6e050e754f9bc4"
server_key_version: "encryptionkey@0"
I20250109 21:07:22.633122 26765 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "fd658876396e4ee6a9097fe40fbb39af"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "cdb29647d56ea3a6c64565f7c376763d"
server_key_iv: "f1aa231fa9758b0f7b6e050e754f9bc4"
server_key_version: "encryptionkey@0"
I20250109 21:07:22.636276 26765 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20250109 21:07:22.638247 26780 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:22.639425 26765 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:07:22.639532 26765 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "fd658876396e4ee6a9097fe40fbb39af"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "cdb29647d56ea3a6c64565f7c376763d"
server_key_iv: "f1aa231fa9758b0f7b6e050e754f9bc4"
server_key_version: "encryptionkey@0"
I20250109 21:07:22.639626 26765 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:22.669857 26765 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:22.672911 26765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:22.673072 26765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:22.673576 26765 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:07:22.674305 26765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:07:22.674352 26765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:22.674412 26765 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:07:22.674438 26765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:22.682608 26765 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:38203
I20250109 21:07:22.682633 26893 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:38203 every 8 connection(s)
I20250109 21:07:22.683531 26765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:07:22.690177 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 26765
I20250109 21:07:22.690312 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:07:22.690600 12793 external_mini_cluster.cc:1453] Setting key e798bc6dff44898cec6f4fdde95c5c17
Jan 09 21:07:22 dist-test-slave-g0zb krb5kdc[26524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456842, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:07:22.695159 26608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:22.685165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:33791 (local address 127.12.126.126:33113)
0109 21:07:22.685372 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:22.685379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:22.686395 (+ 1016us) server_negotiation.cc:407] Connection header received
0109 21:07:22.687154 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:22.687157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:22.687213 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:22.687291 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:22.688417 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.688913 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:22.689479 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.689596 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:22.692013 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:22.692031 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:22.692034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:22.692061 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:22.693447 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:22.693892 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:22.693895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:22.693897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:22.693961 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:22.694207 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:22.694210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:22.694212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:22.694350 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:22.694436 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:22.694889 (+ 453us) server_negotiation.cc:299] Negotiation successful
0109 21:07:22.695012 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20250109 21:07:22.695710 26896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:22.685845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:33791)
0109 21:07:22.686261 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:22.686298 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:22.686971 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:22.687424 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:22.687431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:22.687758 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:22.688268 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:22.688307 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.689015 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:22.689017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:22.689399 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:22.689405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.689561 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:22.690122 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:22.690158 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:22.691857 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:22.693556 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:22.693561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:22.693571 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:22.693799 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:22.694040 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:22.694045 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:22.694047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:22.694124 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:22.694444 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:22.694453 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:22.694656 (+ 203us) client_negotiation.cc:769] Sending connection context
0109 21:07:22.694839 (+ 183us) client_negotiation.cc:240] Negotiation successful
0109 21:07:22.695084 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":123,"threads_started":1}
I20250109 21:07:22.696686 26894 heartbeater.cc:346] Connected to a master server at 127.12.126.126:33113
I20250109 21:07:22.696888 26894 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:22.697338 26894 heartbeater.cc:510] Master 127.12.126.126:33113 requested a full tablet report, sending...
I20250109 21:07:22.698269 26567 ts_manager.cc:194] Registered new tserver with Master: fd658876396e4ee6a9097fe40fbb39af (127.12.126.66:38203)
I20250109 21:07:22.699056 26567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:33791
WARNING: no policy specified for kudu/127.12.126.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
I20250109 21:07:22.715847 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:33113
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:07:22.816903 26899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:22.817101 26899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:22.817175 26899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:22.820256 26899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:22.820312 26899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:22.820430 26899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:07:22.824317 26899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:22.825165 26899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:22.825899 26899 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:22.833860 26905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:22.834581 26907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:22.834846 26904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:22.835686 26899 server_base.cc:1034] running on GCE node
I20250109 21:07:22.836001 26899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:22.836524 26899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:22.837663 26899 hybrid_clock.cc:648] HybridClock initialized: now 1736456842837646 us; error 32 us; skew 500 ppm
Jan 09 21:07:22 dist-test-slave-g0zb krb5kdc[26524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456842, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:22.840166 26899 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:07:22.841022 26899 webserver.cc:458] Webserver started at http://127.12.126.67:35379/ using document root <none> and password file <none>
I20250109 21:07:22.841539 26899 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:22.841598 26899 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:22.841756 26899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:07:22.843478 26899 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "c1ea46aded234e3bbecd1e6237446365"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "484d5a0687f6a34f806aa85bd0c883cb"
server_key_iv: "a4b8f9ca5b38c31294b001e4cadb195c"
server_key_version: "encryptionkey@0"
I20250109 21:07:22.843909 26899 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "c1ea46aded234e3bbecd1e6237446365"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "484d5a0687f6a34f806aa85bd0c883cb"
server_key_iv: "a4b8f9ca5b38c31294b001e4cadb195c"
server_key_version: "encryptionkey@0"
I20250109 21:07:22.847034 26899 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:07:22.849047 26914 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:22.849992 26899 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:07:22.850090 26899 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "c1ea46aded234e3bbecd1e6237446365"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "484d5a0687f6a34f806aa85bd0c883cb"
server_key_iv: "a4b8f9ca5b38c31294b001e4cadb195c"
server_key_version: "encryptionkey@0"
I20250109 21:07:22.850179 26899 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:22.866601 26899 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:22.869021 26899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:22.869176 26899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:22.869643 26899 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:07:22.870352 26899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:07:22.870399 26899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:22.870455 26899 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:07:22.870487 26899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:22.878901 26899 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:40177
I20250109 21:07:22.878933 27027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:40177 every 8 connection(s)
I20250109 21:07:22.879765 26899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:07:22.880666 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 26899
I20250109 21:07:22.880771 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:07:22.881048 12793 external_mini_cluster.cc:1453] Setting key 6267702caddc8965aa408271fae2a9e1
Jan 09 21:07:22 dist-test-slave-g0zb krb5kdc[26524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456842, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:07:22.891891 26608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:22.881504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:45243 (local address 127.12.126.126:33113)
0109 21:07:22.881646 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:22.881650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:22.882985 (+ 1335us) server_negotiation.cc:407] Connection header received
0109 21:07:22.883803 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:22.883808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:22.883872 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:22.883925 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:22.885591 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.886094 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:22.886683 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.886829 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:22.888871 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:22.888884 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:22.888887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:22.888911 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:22.890289 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:22.890770 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:22.890772 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:22.890774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:22.890854 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:22.891104 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:22.891106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:22.891107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:22.891238 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:22.891311 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:22.891690 (+ 379us) server_negotiation.cc:299] Negotiation successful
0109 21:07:22.891778 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250109 21:07:22.892447 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:22.882402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:45243)
0109 21:07:22.882864 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:22.882907 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:22.883585 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:22.884362 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:22.884374 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:22.884918 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:22.885459 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:22.885471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.886202 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:22.886205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:22.886595 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:22.886602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.886766 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:22.887338 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:22.887360 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:22.888735 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:22.890415 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:22.890419 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:22.890427 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:22.890648 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:22.890949 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:22.890951 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:22.890952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:22.891032 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:22.891309 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:22.891313 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:22.891516 (+ 203us) client_negotiation.cc:769] Sending connection context
0109 21:07:22.891662 (+ 146us) client_negotiation.cc:240] Negotiation successful
0109 21:07:22.891845 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":102,"threads_started":1}
I20250109 21:07:22.893388 27028 heartbeater.cc:346] Connected to a master server at 127.12.126.126:33113
I20250109 21:07:22.893607 27028 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:22.894143 27028 heartbeater.cc:510] Master 127.12.126.126:33113 requested a full tablet report, sending...
I20250109 21:07:22.895148 26567 ts_manager.cc:194] Registered new tserver with Master: c1ea46aded234e3bbecd1e6237446365 (127.12.126.67:40177)
I20250109 21:07:22.895749 26567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:45243
I20250109 21:07:22.906138 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 21:07:22.913405 26608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:22.907885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35890 (local address 127.12.126.126:33113)
0109 21:07:22.907987 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:22.907990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:22.908088 (+ 98us) server_negotiation.cc:407] Connection header received
0109 21:07:22.908184 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:22.908186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:22.908221 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:22.908286 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:22.908965 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.909431 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:22.909971 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.910088 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:22.911103 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:22.911115 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:22.911118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:22.911139 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:22.912424 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:22.912699 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:22.912701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:22.912702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:22.912734 (+ 32us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:22.912863 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:22.912865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:22.912866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:22.912995 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:22.913094 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:22.913222 (+ 128us) server_negotiation.cc:299] Negotiation successful
0109 21:07:22.913301 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20250109 21:07:22.915757 26567 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:07:22.918213 26567 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:07:22.930164 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:22.925033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:40177 (local address 127.0.0.1:48918)
0109 21:07:22.925705 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:22.925731 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:22.925855 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:22.926864 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:22.926869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:22.926891 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:22.927179 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:22.927190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.928755 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:22.928759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:22.929686 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:22.929697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.929798 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:22.929851 (+ 53us) client_negotiation.cc:769] Sending connection context
0109 21:07:22.929945 (+ 94us) client_negotiation.cc:240] Negotiation successful
0109 21:07:22.930039 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":63,"threads_started":1}
I20250109 21:07:22.931614 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:22.925154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48918 (local address 127.12.126.67:40177)
0109 21:07:22.925648 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:22.925655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:22.925809 (+ 154us) server_negotiation.cc:407] Connection header received
0109 21:07:22.926133 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:22.926142 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:22.926339 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:22.927356 (+ 1017us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:22.927487 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.928624 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:22.930602 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.931163 (+ 561us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:22.931291 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:22.931362 (+ 71us) server_negotiation.cc:299] Negotiation successful
0109 21:07:22.931454 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":100,"threads_started":1}
I20250109 21:07:22.931942 27039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:22.925397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:38203 (local address 127.0.0.1:47344)
0109 21:07:22.927607 (+ 2210us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:22.927621 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:22.927710 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:22.928501 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:22.928505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:22.928539 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:22.928807 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:22.928815 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.930540 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:22.930543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:22.931493 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:22.931506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.931592 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:22.931608 (+ 16us) client_negotiation.cc:769] Sending connection context
0109 21:07:22.931645 (+ 37us) client_negotiation.cc:240] Negotiation successful
0109 21:07:22.931803 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2131,"thread_start_us":97,"threads_started":1}
I20250109 21:07:22.933315 27044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:22.926551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.12.126.66:38203)
0109 21:07:22.927990 (+ 1439us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:22.927998 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:22.928022 (+ 24us) server_negotiation.cc:407] Connection header received
0109 21:07:22.928115 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:22.928121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:22.928312 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:22.928963 (+ 651us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:22.929144 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.930403 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:22.932367 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.932902 (+ 535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:22.933002 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:22.933077 (+ 75us) server_negotiation.cc:299] Negotiation successful
0109 21:07:22.933168 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1324,"thread_start_us":127,"threads_started":1}
I20250109 21:07:22.934101 27043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:22.925696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41758 (local address 127.12.126.65:34057)
0109 21:07:22.926864 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:22.926872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:22.926899 (+ 27us) server_negotiation.cc:407] Connection header received
0109 21:07:22.926989 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:22.926997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:22.927163 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:22.927347 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:22.927866 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.929627 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:22.932532 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.933354 (+ 822us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:22.933590 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:22.933863 (+ 273us) server_negotiation.cc:299] Negotiation successful
0109 21:07:22.933966 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1023,"thread_start_us":98,"threads_started":1}
I20250109 21:07:22.934535 27040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:22.925157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:34057 (local address 127.0.0.1:41758)
0109 21:07:22.926041 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:22.926057 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:22.926149 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:22.927316 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:22.927318 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:22.927355 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:22.927684 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:22.927693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.930835 (+ 3142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:22.930841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:22.932328 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:22.932340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.933668 (+ 1328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:22.933687 (+ 19us) client_negotiation.cc:769] Sending connection context
0109 21:07:22.934348 (+ 661us) client_negotiation.cc:240] Negotiation successful
0109 21:07:22.934414 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":86,"threads_started":1}
I20250109 21:07:22.934243 26962 tablet_service.cc:1467] Processing CreateTablet for tablet b692d417388146c1bfe4841f4adfc688 (DEFAULT_TABLE table=test-table [id=3fccfc45085341e7b9cb106b5003bfba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:07:22.935168 26962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b692d417388146c1bfe4841f4adfc688. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:22.936131 26828 tablet_service.cc:1467] Processing CreateTablet for tablet b692d417388146c1bfe4841f4adfc688 (DEFAULT_TABLE table=test-table [id=3fccfc45085341e7b9cb106b5003bfba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:07:22.936952 26828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b692d417388146c1bfe4841f4adfc688. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:22.937531 26694 tablet_service.cc:1467] Processing CreateTablet for tablet b692d417388146c1bfe4841f4adfc688 (DEFAULT_TABLE table=test-table [id=3fccfc45085341e7b9cb106b5003bfba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:07:22.938360 26694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b692d417388146c1bfe4841f4adfc688. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:22.940833 27045 tablet_bootstrap.cc:492] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365: Bootstrap starting.
I20250109 21:07:22.942875 27045 tablet_bootstrap.cc:654] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365: Neither blocks nor log segments found. Creating new log.
I20250109 21:07:22.943717 27046 tablet_bootstrap.cc:492] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af: Bootstrap starting.
I20250109 21:07:22.944240 27045 log.cc:826] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:22.944911 27047 tablet_bootstrap.cc:492] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096: Bootstrap starting.
I20250109 21:07:22.945514 27046 tablet_bootstrap.cc:654] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af: Neither blocks nor log segments found. Creating new log.
I20250109 21:07:22.946532 27047 tablet_bootstrap.cc:654] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096: Neither blocks nor log segments found. Creating new log.
I20250109 21:07:22.946525 27046 log.cc:826] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:22.947547 27047 log.cc:826] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:22.948086 27046 tablet_bootstrap.cc:492] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af: No bootstrap required, opened a new log
I20250109 21:07:22.948285 27046 ts_tablet_manager.cc:1397] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250109 21:07:22.949115 27045 tablet_bootstrap.cc:492] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365: No bootstrap required, opened a new log
I20250109 21:07:22.949115 27047 tablet_bootstrap.cc:492] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096: No bootstrap required, opened a new log
I20250109 21:07:22.949265 27045 ts_tablet_manager.cc:1397] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20250109 21:07:22.949265 27047 ts_tablet_manager.cc:1397] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250109 21:07:22.950738 27046 raft_consensus.cc:357] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } } peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } }
I20250109 21:07:22.950951 27046 raft_consensus.cc:383] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:22.951012 27046 raft_consensus.cc:738] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd658876396e4ee6a9097fe40fbb39af, State: Initialized, Role: FOLLOWER
I20250109 21:07:22.951426 27046 consensus_queue.cc:260] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } } peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } }
I20250109 21:07:22.952214 27046 ts_tablet_manager.cc:1428] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250109 21:07:22.952435 27047 raft_consensus.cc:357] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } } peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } }
I20250109 21:07:22.952435 27045 raft_consensus.cc:357] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } } peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } }
I20250109 21:07:22.952597 26894 heartbeater.cc:502] Master 127.12.126.126:33113 was elected leader, sending a full tablet report...
I20250109 21:07:22.952623 27047 raft_consensus.cc:383] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:22.952623 27045 raft_consensus.cc:383] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:22.952672 27047 raft_consensus.cc:738] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59d4d98f404eb6ab752bd586912096, State: Initialized, Role: FOLLOWER
I20250109 21:07:22.952672 27045 raft_consensus.cc:738] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1ea46aded234e3bbecd1e6237446365, State: Initialized, Role: FOLLOWER
I20250109 21:07:22.953065 27045 consensus_queue.cc:260] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } } peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } }
I20250109 21:07:22.953063 27047 consensus_queue.cc:260] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } } peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } }
I20250109 21:07:22.953860 27047 ts_tablet_manager.cc:1428] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20250109 21:07:22.953860 27045 ts_tablet_manager.cc:1428] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:07:22.954206 27028 heartbeater.cc:502] Master 127.12.126.126:33113 was elected leader, sending a full tablet report...
I20250109 21:07:22.954221 26760 heartbeater.cc:502] Master 127.12.126.126:33113 was elected leader, sending a full tablet report...
I20250109 21:07:22.973349 27053 raft_consensus.cc:491] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:07:22.973498 27053 raft_consensus.cc:513] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } } peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } }
I20250109 21:07:22.974561 27053 leader_election.cc:290] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1ea46aded234e3bbecd1e6237446365 (127.12.126.67:40177), fd658876396e4ee6a9097fe40fbb39af (127.12.126.66:38203)
W20250109 21:07:22.975875 26761 tablet.cc:2367] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:07:22.977424 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:22.974748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:40177 (local address 127.12.126.65:39755)
0109 21:07:22.974901 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:22.974913 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:22.974996 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:22.975376 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:22.975379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:22.975400 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:22.975666 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:22.975673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.976495 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:22.976499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:22.977127 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:22.977135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.977224 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:22.977242 (+ 18us) client_negotiation.cc:769] Sending connection context
0109 21:07:22.977277 (+ 35us) client_negotiation.cc:240] Negotiation successful
0109 21:07:22.977326 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20250109 21:07:22.977774 27054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:22.974787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:38203 (local address 127.12.126.65:33001)
0109 21:07:22.975158 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:22.975171 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:22.975273 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:22.975528 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:22.975530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:22.975544 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:22.975779 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:22.975786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.976714 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:22.976717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:22.977330 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:22.977340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.977444 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:22.977515 (+ 71us) client_negotiation.cc:769] Sending connection context
0109 21:07:22.977569 (+ 54us) client_negotiation.cc:240] Negotiation successful
0109 21:07:22.977625 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":123,"threads_started":1}
I20250109 21:07:22.977986 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:22.975029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:39755 (local address 127.12.126.67:40177)
0109 21:07:22.975153 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:22.975157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:22.975173 (+ 16us) server_negotiation.cc:407] Connection header received
0109 21:07:22.975228 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:22.975230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:22.975279 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:22.975356 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:22.975790 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.976389 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:22.977248 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.977710 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:22.977748 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:22.977800 (+ 52us) server_negotiation.cc:299] Negotiation successful
0109 21:07:22.977860 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250109 21:07:22.978202 27044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:22.974905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:33001 (local address 127.12.126.66:38203)
0109 21:07:22.975050 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:22.975058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:22.975246 (+ 188us) server_negotiation.cc:407] Connection header received
0109 21:07:22.975378 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:22.975381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:22.975425 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:22.975504 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:22.975901 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.976609 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:22.977460 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:22.977950 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:22.977983 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:22.978027 (+ 44us) server_negotiation.cc:299] Negotiation successful
0109 21:07:22.978094 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20250109 21:07:22.978650 26982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b692d417388146c1bfe4841f4adfc688" candidate_uuid: "cd59d4d98f404eb6ab752bd586912096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1ea46aded234e3bbecd1e6237446365" is_pre_election: true
I20250109 21:07:22.978758 26848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b692d417388146c1bfe4841f4adfc688" candidate_uuid: "cd59d4d98f404eb6ab752bd586912096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd658876396e4ee6a9097fe40fbb39af" is_pre_election: true
I20250109 21:07:22.978929 26982 raft_consensus.cc:2463] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd59d4d98f404eb6ab752bd586912096 in term 0.
I20250109 21:07:22.979044 26848 raft_consensus.cc:2463] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd59d4d98f404eb6ab752bd586912096 in term 0.
I20250109 21:07:22.979383 26647 leader_election.cc:304] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1ea46aded234e3bbecd1e6237446365, cd59d4d98f404eb6ab752bd586912096; no voters:
I20250109 21:07:22.979627 27053 raft_consensus.cc:2798] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:07:22.979698 27053 raft_consensus.cc:491] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:07:22.979728 27053 raft_consensus.cc:3054] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:22.980660 27053 raft_consensus.cc:513] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } } peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } }
I20250109 21:07:22.980980 27053 leader_election.cc:290] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 1 election: Requested vote from peers c1ea46aded234e3bbecd1e6237446365 (127.12.126.67:40177), fd658876396e4ee6a9097fe40fbb39af (127.12.126.66:38203)
I20250109 21:07:22.981297 26848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b692d417388146c1bfe4841f4adfc688" candidate_uuid: "cd59d4d98f404eb6ab752bd586912096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd658876396e4ee6a9097fe40fbb39af"
I20250109 21:07:22.981297 26982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b692d417388146c1bfe4841f4adfc688" candidate_uuid: "cd59d4d98f404eb6ab752bd586912096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1ea46aded234e3bbecd1e6237446365"
I20250109 21:07:22.981410 26848 raft_consensus.cc:3054] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:22.981418 26982 raft_consensus.cc:3054] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:22.982545 26848 raft_consensus.cc:2463] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59d4d98f404eb6ab752bd586912096 in term 1.
I20250109 21:07:22.982545 26982 raft_consensus.cc:2463] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59d4d98f404eb6ab752bd586912096 in term 1.
I20250109 21:07:22.982851 26647 leader_election.cc:304] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1ea46aded234e3bbecd1e6237446365, cd59d4d98f404eb6ab752bd586912096; no voters:
I20250109 21:07:22.983024 27053 raft_consensus.cc:2798] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:07:22.983223 27053 raft_consensus.cc:695] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [term 1 LEADER]: Becoming Leader. State: Replica: cd59d4d98f404eb6ab752bd586912096, State: Running, Role: LEADER
I20250109 21:07:22.983502 27053 consensus_queue.cc:237] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } } peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } }
I20250109 21:07:22.986274 26567 catalog_manager.cc:5526] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd59d4d98f404eb6ab752bd586912096 (127.12.126.65). New cstate: current_term: 1 leader_uuid: "cd59d4d98f404eb6ab752bd586912096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } health_report { overall_health: UNKNOWN } } }
I20250109 21:07:23.009250 27043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.006565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.12.126.65:34057)
0109 21:07:23.006699 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:23.006702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:23.006731 (+ 29us) server_negotiation.cc:407] Connection header received
0109 21:07:23.006803 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:23.006806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:23.006899 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:23.006991 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:07:23.007262 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.007779 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.008370 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.008503 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.008584 (+ 81us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:07:23.008983 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:07:23.009075 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:23.009105 (+ 30us) server_negotiation.cc:299] Negotiation successful
0109 21:07:23.009151 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250109 21:07:23.014355 26848 raft_consensus.cc:1270] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af [term 1 FOLLOWER]: Refusing update from remote peer cd59d4d98f404eb6ab752bd586912096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:07:23.014354 26982 raft_consensus.cc:1270] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365 [term 1 FOLLOWER]: Refusing update from remote peer cd59d4d98f404eb6ab752bd586912096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:07:23.015043 27055 consensus_queue.cc:1035] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:07:23.015192 27053 consensus_queue.cc:1035] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:07:23.021112 27058 mvcc.cc:204] Tried to move back new op lower bound from 7112527228980256768 to 7112527228862648320. Current Snapshot: MvccSnapshot[applied={T|T < 7112527228980256768}]
I20250109 21:07:23.021878 27060 mvcc.cc:204] Tried to move back new op lower bound from 7112527228980256768 to 7112527228862648320. Current Snapshot: MvccSnapshot[applied={T|T < 7112527228980256768}]
I20250109 21:07:23.024106 27059 mvcc.cc:204] Tried to move back new op lower bound from 7112527228980256768 to 7112527228862648320. Current Snapshot: MvccSnapshot[applied={T|T < 7112527228980256768}]
I20250109 21:07:23.031054 26568 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35890:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:07:23.031188 26568 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35890:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:07:23.033293 26568 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f: Sending DeleteTablet for 3 replicas of tablet b692d417388146c1bfe4841f4adfc688
I20250109 21:07:23.033932 26694 tablet_service.cc:1514] Processing DeleteTablet for tablet b692d417388146c1bfe4841f4adfc688 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:07:23 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:41758
I20250109 21:07:23.034024 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 26631
I20250109 21:07:23.033989 26962 tablet_service.cc:1514] Processing DeleteTablet for tablet b692d417388146c1bfe4841f4adfc688 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:07:23 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:48918
I20250109 21:07:23.035702 27067 tablet_replica.cc:331] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365: stopping tablet replica
I20250109 21:07:23.035729 26828 tablet_service.cc:1514] Processing DeleteTablet for tablet b692d417388146c1bfe4841f4adfc688 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:07:23 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:47344
I20250109 21:07:23.035990 27067 raft_consensus.cc:2238] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:07:23.036127 27068 tablet_replica.cc:331] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af: stopping tablet replica
I20250109 21:07:23.036340 27068 raft_consensus.cc:2238] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:07:23.036348 27067 raft_consensus.cc:2267] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:07:23.036613 27068 raft_consensus.cc:2267] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:07:23.037667 27068 ts_tablet_manager.cc:1905] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:07:23.037717 27067 ts_tablet_manager.cc:1905] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:07:23.039621 27068 ts_tablet_manager.cc:1918] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:07:23.039705 27068 log.cc:1198] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/b692d417388146c1bfe4841f4adfc688
I20250109 21:07:23.040041 27068 ts_tablet_manager.cc:1939] T b692d417388146c1bfe4841f4adfc688 P fd658876396e4ee6a9097fe40fbb39af: Deleting consensus metadata
I20250109 21:07:23.040696 27067 ts_tablet_manager.cc:1918] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:07:23.040783 27067 log.cc:1198] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/b692d417388146c1bfe4841f4adfc688
I20250109 21:07:23.040951 26556 catalog_manager.cc:4872] TS fd658876396e4ee6a9097fe40fbb39af (127.12.126.66:38203): tablet b692d417388146c1bfe4841f4adfc688 (table test-table [id=3fccfc45085341e7b9cb106b5003bfba]) successfully deleted
I20250109 21:07:23.041141 27067 ts_tablet_manager.cc:1939] T b692d417388146c1bfe4841f4adfc688 P c1ea46aded234e3bbecd1e6237446365: Deleting consensus metadata
W20250109 21:07:23.041575 26554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.126.65:34057 (error 108)
I20250109 21:07:23.041826 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 26765
I20250109 21:07:23.043565 26553 catalog_manager.cc:4872] TS c1ea46aded234e3bbecd1e6237446365 (127.12.126.67:40177): tablet b692d417388146c1bfe4841f4adfc688 (table test-table [id=3fccfc45085341e7b9cb106b5003bfba]) successfully deleted
I20250109 21:07:23.044575 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.044340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:34057 (local address 127.0.0.1:41778)
0109 21:07:23.044461 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:23.044513 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.65:34057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20250109 21:07:23.044715 26554 catalog_manager.cc:4615] TS cd59d4d98f404eb6ab752bd586912096 (127.12.126.65:34057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b692d417388146c1bfe4841f4adfc688: Network error: Client connection negotiation failed: client connection to 127.12.126.65:34057: connect: Connection refused (error 111)
I20250109 21:07:23.047147 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 26899
I20250109 21:07:23.052163 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 26537
I20250109 21:07:23.058563 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:33113
--webserver_interface=127.12.126.126
--webserver_port=46511
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:33113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:07:23.159900 27070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:23.160118 27070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:23.160180 27070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:23.163133 27070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:07:23.163179 27070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:23.163200 27070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:23.163233 27070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:07:23.163266 27070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:07:23.167194 27070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:33113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:33113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=46511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:23.168087 27070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:23.168859 27070 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:23.179230 27079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:23.179901 27076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:23.180774 27077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:23.181674 27070 server_base.cc:1034] running on GCE node
I20250109 21:07:23.182163 27070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:23.183130 27070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:23.184294 27070 hybrid_clock.cc:648] HybridClock initialized: now 1736456843184282 us; error 31 us; skew 500 ppm
Jan 09 21:07:23 dist-test-slave-g0zb krb5kdc[26524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456843, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:23.186978 27070 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:07:23.187835 27070 webserver.cc:458] Webserver started at http://127.12.126.126:46511/ using document root <none> and password file <none>
I20250109 21:07:23.188385 27070 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:23.188455 27070 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:23.191870 27070 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20250109 21:07:23.193622 27086 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:23.194514 27070 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:07:23.194620 27070 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "72863e5fa41f5e8b9c842c8f307bad38"
server_key_iv: "f06545f6132740a1c757654e97bd4ae3"
server_key_version: "encryptionkey@0"
I20250109 21:07:23.195101 27070 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:23.204594 27070 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:23.207054 27070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:23.207284 27070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:23.213603 27070 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:33113
I20250109 21:07:23.213840 27138 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:33113 every 8 connection(s)
I20250109 21:07:23.214663 27070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:07:23.218403 27139 sys_catalog.cc:263] Verifying existing consensus state
I20250109 21:07:23.219918 27139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f: Bootstrap starting.
I20250109 21:07:23.223320 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 27070
I20250109 21:07:23.223717 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:34057
--local_ip_for_outbound_sockets=127.12.126.65
--tserver_master_addrs=127.12.126.126:33113
--webserver_port=35675
--webserver_interface=127.12.126.65
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:07:23.235754 27139 log.cc:826] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:23.243732 27139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:07:23.244086 27139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f: Bootstrap complete.
I20250109 21:07:23.247701 27139 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33113 } }
I20250109 21:07:23.248003 27139 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0e9f0299b74ce2ba7c731714c2cf6f, State: Initialized, Role: FOLLOWER
I20250109 21:07:23.248477 27139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33113 } }
I20250109 21:07:23.248600 27139 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:07:23.248664 27139 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:07:23.248766 27139 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:07:23.250272 27139 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33113 } }
I20250109 21:07:23.250582 27139 leader_election.cc:304] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f0e9f0299b74ce2ba7c731714c2cf6f; no voters:
I20250109 21:07:23.250876 27139 leader_election.cc:290] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [CANDIDATE]: Term 2 election: Requested vote from peers
I20250109 21:07:23.250960 27143 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 2 FOLLOWER]: Leader election won for term 2
I20250109 21:07:23.251155 27143 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 2 LEADER]: Becoming Leader. State: Replica: 8f0e9f0299b74ce2ba7c731714c2cf6f, State: Running, Role: LEADER
I20250109 21:07:23.251482 27143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33113 } }
I20250109 21:07:23.251967 27139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:07:23.252736 27144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33113 } } }
I20250109 21:07:23.252847 27144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [sys.catalog]: This master's current role is: LEADER
I20250109 21:07:23.253181 27145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f0e9f0299b74ce2ba7c731714c2cf6f. Latest consensus state: current_term: 2 leader_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33113 } } }
I20250109 21:07:23.253229 27151 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:07:23.253257 27145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [sys.catalog]: This master's current role is: LEADER
I20250109 21:07:23.257601 27151 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:07:23.258291 27151 catalog_manager.cc:1260] Loaded cluster ID: ee0104f07acb45af8684c5ff9af5d1c7
I20250109 21:07:23.258351 27151 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:07:23.259735 27151 catalog_manager.cc:1315] IPKI private key was stored in cleartext, attempting to encrypt it.
I20250109 21:07:23.262681 27151 catalog_manager.cc:1325] IPKI private key successfully encrypted.
I20250109 21:07:23.267671 27151 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:07:23.268394 27151 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f: Loaded TSK: 0
I20250109 21:07:23.269537 27151 catalog_manager.cc:1515] Initializing in-progress tserver states...
W20250109 21:07:23.327162 27141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:23.327368 27141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:23.327445 27141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:23.330459 27141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:23.330513 27141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:23.330607 27141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:07:23.334435 27141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:34057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=35675
--tserver_master_addrs=127.12.126.126:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:23.335352 27141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:23.336093 27141 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:23.343727 27166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:23.344410 27169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:23.345258 27167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:23.346081 27141 server_base.cc:1034] running on GCE node
I20250109 21:07:23.346454 27141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:23.347047 27141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:23.348253 27141 hybrid_clock.cc:648] HybridClock initialized: now 1736456843348236 us; error 32 us; skew 500 ppm
Jan 09 21:07:23 dist-test-slave-g0zb krb5kdc[26524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456843, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:23.351332 27141 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:07:23.352259 27141 webserver.cc:458] Webserver started at http://127.12.126.65:35675/ using document root <none> and password file <none>
I20250109 21:07:23.352856 27141 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:23.353056 27141 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:23.357072 27141 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:07:23.359256 27176 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:23.360419 27141 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:07:23.360523 27141 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "cd59d4d98f404eb6ab752bd586912096"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "e3749b2e75136e80788ab3049b38d2ef"
server_key_iv: "6e7ca46d61ba11a4bfc6aa76fc8d3865"
server_key_version: "encryptionkey@0"
I20250109 21:07:23.360980 27141 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:23.380365 27141 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:23.383528 27141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:23.383692 27141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:23.384263 27141 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:07:23.385401 27183 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250109 21:07:23.388762 27141 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250109 21:07:23.388829 27141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20250109 21:07:23.388888 27141 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250109 21:07:23.390843 27141 ts_tablet_manager.cc:610] Registered 1 tablets
I20250109 21:07:23.390901 27141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20250109 21:07:23.391278 27183 tablet_bootstrap.cc:492] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096: Bootstrap starting.
I20250109 21:07:23.401222 27141 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:34057
I20250109 21:07:23.401448 27290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:34057 every 8 connection(s)
I20250109 21:07:23.402411 27141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:07:23.409035 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 27141
I20250109 21:07:23.409533 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:38203
--local_ip_for_outbound_sockets=127.12.126.66
--tserver_master_addrs=127.12.126.126:33113
--webserver_port=37509
--webserver_interface=127.12.126.66
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:07:23.410512 27183 log.cc:826] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096: Log is configured to *not* fsync() on all Append() calls
Jan 09 21:07:23 dist-test-slave-g0zb krb5kdc[26524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456843, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:07:23.419806 27294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.404354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:35473)
0109 21:07:23.405117 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:23.405161 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:23.406063 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:23.407230 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:23.407239 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:23.407720 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:23.408539 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:23.408556 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.410142 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.410146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:23.410735 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:23.410743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.411376 (+ 633us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.412043 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:23.412074 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:23.413945 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:23.416915 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:23.416922 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:23.416936 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:23.417281 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:23.417631 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:23.417634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:23.417636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:23.417742 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:23.418327 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:23.418333 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:23.418586 (+ 253us) client_negotiation.cc:769] Sending connection context
0109 21:07:23.418781 (+ 195us) client_negotiation.cc:240] Negotiation successful
0109 21:07:23.419058 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":119,"threads_started":1}
I20250109 21:07:23.419834 27293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.404414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:35473 (local address 127.12.126.126:33113)
0109 21:07:23.404802 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:23.404814 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:23.405302 (+ 488us) server_negotiation.cc:407] Connection header received
0109 21:07:23.406481 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:23.406495 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:23.406785 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:23.407166 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:23.408701 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.409980 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.410889 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.411147 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.414156 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:23.414207 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:23.414224 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:23.414272 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:23.416775 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:23.417419 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:23.417427 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:23.417432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:23.417526 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:23.417841 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:23.417844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:23.417846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:23.418206 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:23.418339 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:23.418941 (+ 602us) server_negotiation.cc:299] Negotiation successful
0109 21:07:23.419206 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":93,"threads_started":1}
I20250109 21:07:23.423136 27291 heartbeater.cc:346] Connected to a master server at 127.12.126.126:33113
I20250109 21:07:23.423360 27291 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:23.423388 27183 tablet_bootstrap.cc:492] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:07:23.423709 27183 tablet_bootstrap.cc:492] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096: Bootstrap complete.
I20250109 21:07:23.423873 27291 heartbeater.cc:510] Master 127.12.126.126:33113 requested a full tablet report, sending...
I20250109 21:07:23.424746 27183 ts_tablet_manager.cc:1397] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096: Time spent bootstrapping tablet: real 0.034s user 0.018s sys 0.008s
I20250109 21:07:23.426283 27103 ts_manager.cc:194] Registered new tserver with Master: cd59d4d98f404eb6ab752bd586912096 (127.12.126.65:34057)
I20250109 21:07:23.427014 27183 raft_consensus.cc:357] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } } peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } }
I20250109 21:07:23.427453 27183 raft_consensus.cc:738] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59d4d98f404eb6ab752bd586912096, State: Initialized, Role: FOLLOWER
I20250109 21:07:23.430145 27103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:35473
I20250109 21:07:23.433925 27183 consensus_queue.cc:260] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } } peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } }
I20250109 21:07:23.434536 27291 heartbeater.cc:502] Master 127.12.126.126:33113 was elected leader, sending a full tablet report...
I20250109 21:07:23.434767 27183 ts_tablet_manager.cc:1428] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250109 21:07:23.435762 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.429393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:34057 (local address 127.0.0.1:41782)
0109 21:07:23.430622 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:23.430653 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:23.430767 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:23.432939 (+ 2172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:23.432944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:23.432964 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:23.433283 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:23.433294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.434485 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.434488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:23.435309 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:23.435318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.435421 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.435464 (+ 43us) client_negotiation.cc:769] Sending connection context
0109 21:07:23.435550 (+ 86us) client_negotiation.cc:240] Negotiation successful
0109 21:07:23.435632 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1094,"thread_start_us":115,"threads_started":1}
I20250109 21:07:23.436255 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.430054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41782 (local address 127.12.126.65:34057)
0109 21:07:23.430404 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:23.430409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:23.430970 (+ 561us) server_negotiation.cc:407] Connection header received
0109 21:07:23.431137 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:23.431142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:23.432654 (+ 1512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:23.432800 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:23.433418 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.434360 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.435428 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.435899 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.435998 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:23.436064 (+ 66us) server_negotiation.cc:299] Negotiation successful
0109 21:07:23.436141 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":87,"threads_started":1}
I20250109 21:07:23.437033 27223 tablet_service.cc:1514] Processing DeleteTablet for tablet b692d417388146c1bfe4841f4adfc688 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:07:23 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:41782
I20250109 21:07:23.437757 27300 tablet_replica.cc:331] stopping tablet replica
I20250109 21:07:23.437974 27300 raft_consensus.cc:2238] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:07:23.438180 27300 raft_consensus.cc:2267] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:07:23.438997 27300 ts_tablet_manager.cc:1905] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:07:23.441953 27300 ts_tablet_manager.cc:1918] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:07:23.442023 27300 log.cc:1198] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/wal/wals/b692d417388146c1bfe4841f4adfc688
I20250109 21:07:23.442312 27300 ts_tablet_manager.cc:1939] T b692d417388146c1bfe4841f4adfc688 P cd59d4d98f404eb6ab752bd586912096: Deleting consensus metadata
I20250109 21:07:23.443668 27088 catalog_manager.cc:4872] TS cd59d4d98f404eb6ab752bd586912096 (127.12.126.65:34057): tablet b692d417388146c1bfe4841f4adfc688 (table test-table [id=3fccfc45085341e7b9cb106b5003bfba]) successfully deleted
W20250109 21:07:23.524864 27296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:23.525079 27296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:23.525152 27296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:23.528102 27296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:23.528177 27296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:23.528280 27296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:07:23.532178 27296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:38203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=37509
--tserver_master_addrs=127.12.126.126:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:23.533102 27296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:23.533860 27296 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:23.541357 27305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:23.542124 27306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:23.542867 27308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:23.543744 27296 server_base.cc:1034] running on GCE node
I20250109 21:07:23.544083 27296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:23.544554 27296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:23.545709 27296 hybrid_clock.cc:648] HybridClock initialized: now 1736456843545700 us; error 24 us; skew 500 ppm
Jan 09 21:07:23 dist-test-slave-g0zb krb5kdc[26524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456843, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:23.548247 27296 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:07:23.549017 27296 webserver.cc:458] Webserver started at http://127.12.126.66:37509/ using document root <none> and password file <none>
I20250109 21:07:23.549540 27296 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:23.549607 27296 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:23.552732 27296 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20250109 21:07:23.554488 27315 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:23.555444 27296 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:07:23.555547 27296 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "fd658876396e4ee6a9097fe40fbb39af"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "cdb29647d56ea3a6c64565f7c376763d"
server_key_iv: "f1aa231fa9758b0f7b6e050e754f9bc4"
server_key_version: "encryptionkey@0"
I20250109 21:07:23.555912 27296 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:23.578361 27296 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:23.581889 27296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:23.582091 27296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:23.582692 27296 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:07:23.583628 27296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:07:23.583683 27296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:23.583739 27296 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:07:23.583768 27296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:23.592391 27296 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:38203
I20250109 21:07:23.592417 27428 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:38203 every 8 connection(s)
I20250109 21:07:23.593334 27296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:07:23.595398 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 27296
I20250109 21:07:23.595849 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:40177
--local_ip_for_outbound_sockets=127.12.126.67
--tserver_master_addrs=127.12.126.126:33113
--webserver_port=35379
--webserver_interface=127.12.126.67
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 09 21:07:23 dist-test-slave-g0zb krb5kdc[26524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456843, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:07:23.605144 27293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.594897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:47273 (local address 127.12.126.126:33113)
0109 21:07:23.595033 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:23.595037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:23.596276 (+ 1239us) server_negotiation.cc:407] Connection header received
0109 21:07:23.597006 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:23.597010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:23.597070 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:23.597145 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:23.598276 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.598752 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.599455 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.599716 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.601528 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:23.601552 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:23.601557 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:23.601595 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:23.603334 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:23.603773 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:23.603778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:23.603783 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:23.603854 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:23.604098 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:23.604101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:23.604104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:23.604295 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:23.604404 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:23.604819 (+ 415us) server_negotiation.cc:299] Negotiation successful
0109 21:07:23.604950 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250109 21:07:23.605557 27431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.595678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:47273)
0109 21:07:23.596145 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:23.596189 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:23.596819 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:23.597299 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:23.597306 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:23.597627 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:23.598135 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:23.598146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.598913 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.598916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:23.599336 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:23.599345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.599505 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.600010 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:23.600033 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:23.601377 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:23.603448 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:23.603452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:23.603460 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:23.603678 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:23.603938 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:23.603941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:23.603942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:23.604013 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:23.604386 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:23.604389 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:23.604597 (+ 208us) client_negotiation.cc:769] Sending connection context
0109 21:07:23.604743 (+ 146us) client_negotiation.cc:240] Negotiation successful
0109 21:07:23.604929 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":119,"threads_started":1}
I20250109 21:07:23.606608 27429 heartbeater.cc:346] Connected to a master server at 127.12.126.126:33113
I20250109 21:07:23.606839 27429 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:23.607373 27429 heartbeater.cc:510] Master 127.12.126.126:33113 requested a full tablet report, sending...
I20250109 21:07:23.608454 27103 ts_manager.cc:194] Registered new tserver with Master: fd658876396e4ee6a9097fe40fbb39af (127.12.126.66:38203)
I20250109 21:07:23.609020 27103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:47273
W20250109 21:07:23.697569 27432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:23.697763 27432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:23.697814 27432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:23.700946 27432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:23.700999 27432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:23.701095 27432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:07:23.704870 27432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:40177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=35379
--tserver_master_addrs=127.12.126.126:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:23.705790 27432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:23.706591 27432 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:23.714776 27437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:23.715327 27438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:23.716161 27440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:23.717033 27432 server_base.cc:1034] running on GCE node
I20250109 21:07:23.717370 27432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:23.717836 27432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:23.718971 27432 hybrid_clock.cc:648] HybridClock initialized: now 1736456843718962 us; error 21 us; skew 500 ppm
Jan 09 21:07:23 dist-test-slave-g0zb krb5kdc[26524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456843, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:23.721428 27432 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:07:23.722261 27432 webserver.cc:458] Webserver started at http://127.12.126.67:35379/ using document root <none> and password file <none>
I20250109 21:07:23.722760 27432 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:23.722852 27432 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:23.726074 27432 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:07:23.727790 27447 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:23.728633 27432 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:07:23.728729 27432 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "c1ea46aded234e3bbecd1e6237446365"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "484d5a0687f6a34f806aa85bd0c883cb"
server_key_iv: "a4b8f9ca5b38c31294b001e4cadb195c"
server_key_version: "encryptionkey@0"
I20250109 21:07:23.729104 27432 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:23.748474 27432 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:23.750816 27432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:23.750991 27432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:23.751446 27432 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:07:23.752137 27432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:07:23.752182 27432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:23.752259 27432 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:07:23.752300 27432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:23.761888 27432 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:40177
I20250109 21:07:23.761931 27560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:40177 every 8 connection(s)
I20250109 21:07:23.762919 27432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
Jan 09 21:07:23 dist-test-slave-g0zb krb5kdc[26524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456843, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:07:23.771260 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 27432
I20250109 21:07:23.774312 27293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.764487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:36147 (local address 127.12.126.126:33113)
0109 21:07:23.764615 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:23.764619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:23.765675 (+ 1056us) server_negotiation.cc:407] Connection header received
0109 21:07:23.766417 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:23.766420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:23.766465 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:23.766534 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:23.767722 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.768199 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.768719 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.768833 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.770796 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:23.770835 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:23.770839 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:23.770862 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:23.772637 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:23.773054 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:23.773057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:23.773060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:23.773115 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:23.773355 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:23.773361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:23.773365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:23.773504 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:23.773577 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:23.774049 (+ 472us) server_negotiation.cc:299] Negotiation successful
0109 21:07:23.774160 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250109 21:07:23.774909 27563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.765124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:36147)
0109 21:07:23.765559 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:23.765592 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:23.766225 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:23.766659 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:23.766666 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:23.767065 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:23.767604 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:23.767616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.768286 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.768289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:23.768642 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:23.768648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.768794 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.769285 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:23.769305 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:23.770670 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:23.772740 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:23.772744 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:23.772751 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:23.772958 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:23.773202 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:23.773204 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:23.773205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:23.773277 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:23.773591 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:23.773596 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:23.773834 (+ 238us) client_negotiation.cc:769] Sending connection context
0109 21:07:23.774027 (+ 193us) client_negotiation.cc:240] Negotiation successful
0109 21:07:23.774247 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":102,"threads_started":1}
I20250109 21:07:23.775914 27561 heartbeater.cc:346] Connected to a master server at 127.12.126.126:33113
I20250109 21:07:23.776118 27561 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:23.776618 27561 heartbeater.cc:510] Master 127.12.126.126:33113 requested a full tablet report, sending...
I20250109 21:07:23.777642 27103 ts_manager.cc:194] Registered new tserver with Master: c1ea46aded234e3bbecd1e6237446365 (127.12.126.67:40177)
I20250109 21:07:23.778286 27103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:36147
I20250109 21:07:23.778893 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.771828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33686 (local address 127.12.126.126:33113)
0109 21:07:23.772040 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:23.772043 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:23.772182 (+ 139us) server_negotiation.cc:407] Connection header received
0109 21:07:23.772341 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:23.772343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:23.772375 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:23.772442 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:23.773382 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.774022 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.774760 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.774968 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.776029 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:23.776047 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:23.776050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:23.776072 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:23.777605 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:23.777962 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:23.777965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:23.777967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:23.778018 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:23.778191 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:23.778193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:23.778195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:23.778321 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:23.778482 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:23.778621 (+ 139us) server_negotiation.cc:299] Negotiation successful
0109 21:07:23.778732 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":77,"threads_started":1}
I20250109 21:07:23.780283 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 21:07:23.783525 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.780762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33694 (local address 127.12.126.126:33113)
0109 21:07:23.780868 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:23.780872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:23.780988 (+ 116us) server_negotiation.cc:407] Connection header received
0109 21:07:23.781123 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:23.781126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:23.781190 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:23.781282 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:07:23.781580 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.782036 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.782625 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.782784 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.782904 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:07:23.783213 (+ 309us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:07:23.783317 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:23.783392 (+ 75us) server_negotiation.cc:299] Negotiation successful
0109 21:07:23.783434 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250109 21:07:23.784030 27103 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:07:23.786530 27103 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:07:23.792802 27223 tablet_service.cc:1467] Processing CreateTablet for tablet 134dac1af42f4c939db1688ba295eb1f (DEFAULT_TABLE table=test-table [id=2482343db42149a89dc2349d345dbbf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:07:23.793386 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 134dac1af42f4c939db1688ba295eb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:23.795814 27183 tablet_bootstrap.cc:492] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096: Bootstrap starting.
I20250109 21:07:23.795833 27570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.792187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:40177 (local address 127.0.0.1:36946)
0109 21:07:23.792565 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:23.792578 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:23.792681 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:23.793234 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:23.793243 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:23.793259 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:23.793449 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:23.793461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.794496 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.794499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:23.795482 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:23.795491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.795596 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.795611 (+ 15us) client_negotiation.cc:769] Sending connection context
0109 21:07:23.795652 (+ 41us) client_negotiation.cc:240] Negotiation successful
0109 21:07:23.795704 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":90,"threads_started":1}
I20250109 21:07:23.796047 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.791854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:38203 (local address 127.0.0.1:51416)
0109 21:07:23.791958 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:23.791971 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:23.792038 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:23.793268 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:23.793272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:23.793305 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:23.793493 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:23.793501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.794962 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.794966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:23.795724 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:23.795731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.795815 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.795828 (+ 13us) client_negotiation.cc:769] Sending connection context
0109 21:07:23.795875 (+ 47us) client_negotiation.cc:240] Negotiation successful
0109 21:07:23.795932 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20250109 21:07:23.796497 27572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.792530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36946 (local address 127.12.126.67:40177)
0109 21:07:23.792861 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:23.792864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:23.792880 (+ 16us) server_negotiation.cc:407] Connection header received
0109 21:07:23.792931 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:23.792935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:23.793096 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:23.793244 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:23.793581 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.794270 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.795635 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.796105 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.796211 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:23.796290 (+ 79us) server_negotiation.cc:299] Negotiation successful
0109 21:07:23.796370 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":126,"threads_started":1}
I20250109 21:07:23.796648 27183 tablet_bootstrap.cc:654] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096: Neither blocks nor log segments found. Creating new log.
I20250109 21:07:23.797842 27183 tablet_bootstrap.cc:492] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096: No bootstrap required, opened a new log
I20250109 21:07:23.797843 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.792156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.12.126.66:38203)
0109 21:07:23.792792 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:23.792798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:23.792822 (+ 24us) server_negotiation.cc:407] Connection header received
0109 21:07:23.792909 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:23.792914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:23.793084 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:23.793644 (+ 560us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:23.793778 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.794867 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.796397 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.797485 (+ 1088us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.797582 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:23.797644 (+ 62us) server_negotiation.cc:299] Negotiation successful
0109 21:07:23.797722 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":111,"threads_started":1}
I20250109 21:07:23.797922 27183 ts_tablet_manager.cc:1397] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250109 21:07:23.798539 27183 raft_consensus.cc:357] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } } peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } }
I20250109 21:07:23.798664 27183 raft_consensus.cc:383] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:23.798702 27183 raft_consensus.cc:738] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59d4d98f404eb6ab752bd586912096, State: Initialized, Role: FOLLOWER
I20250109 21:07:23.798846 27183 consensus_queue.cc:260] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } } peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } }
I20250109 21:07:23.799099 27183 ts_tablet_manager.cc:1428] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 21:07:23.798928 27495 tablet_service.cc:1467] Processing CreateTablet for tablet 134dac1af42f4c939db1688ba295eb1f (DEFAULT_TABLE table=test-table [id=2482343db42149a89dc2349d345dbbf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:07:23.799801 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 134dac1af42f4c939db1688ba295eb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:23.800035 27363 tablet_service.cc:1467] Processing CreateTablet for tablet 134dac1af42f4c939db1688ba295eb1f (DEFAULT_TABLE table=test-table [id=2482343db42149a89dc2349d345dbbf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:07:23.800660 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 134dac1af42f4c939db1688ba295eb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:23.805003 27574 tablet_bootstrap.cc:492] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365: Bootstrap starting.
I20250109 21:07:23.805811 27575 tablet_bootstrap.cc:492] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af: Bootstrap starting.
I20250109 21:07:23.807094 27574 tablet_bootstrap.cc:654] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365: Neither blocks nor log segments found. Creating new log.
I20250109 21:07:23.807809 27575 tablet_bootstrap.cc:654] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af: Neither blocks nor log segments found. Creating new log.
I20250109 21:07:23.808266 27574 log.cc:826] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:23.808887 27575 log.cc:826] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:23.809934 27574 tablet_bootstrap.cc:492] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365: No bootstrap required, opened a new log
I20250109 21:07:23.810117 27574 ts_tablet_manager.cc:1397] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:07:23.810493 27575 tablet_bootstrap.cc:492] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af: No bootstrap required, opened a new log
I20250109 21:07:23.810673 27575 ts_tablet_manager.cc:1397] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:07:23.813274 27574 raft_consensus.cc:357] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } } peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } }
I20250109 21:07:23.813472 27574 raft_consensus.cc:383] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:23.813519 27574 raft_consensus.cc:738] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1ea46aded234e3bbecd1e6237446365, State: Initialized, Role: FOLLOWER
I20250109 21:07:23.813628 27575 raft_consensus.cc:357] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } } peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } }
I20250109 21:07:23.813843 27575 raft_consensus.cc:383] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:23.813896 27575 raft_consensus.cc:738] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd658876396e4ee6a9097fe40fbb39af, State: Initialized, Role: FOLLOWER
I20250109 21:07:23.813944 27574 consensus_queue.cc:260] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } } peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } }
I20250109 21:07:23.814303 27575 consensus_queue.cc:260] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } } peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } }
I20250109 21:07:23.814690 27561 heartbeater.cc:502] Master 127.12.126.126:33113 was elected leader, sending a full tablet report...
I20250109 21:07:23.814770 27574 ts_tablet_manager.cc:1428] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:07:23.815042 27575 ts_tablet_manager.cc:1428] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20250109 21:07:23.815328 27429 heartbeater.cc:502] Master 127.12.126.126:33113 was elected leader, sending a full tablet report...
I20250109 21:07:23.841545 27579 raft_consensus.cc:491] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:07:23.841686 27579 raft_consensus.cc:513] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } } peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } }
I20250109 21:07:23.842535 27579 leader_election.cc:290] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1ea46aded234e3bbecd1e6237446365 (127.12.126.67:40177), cd59d4d98f404eb6ab752bd586912096 (127.12.126.65:34057)
W20250109 21:07:23.844128 27430 tablet.cc:2367] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:07:23.845328 27431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.842722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:40177 (local address 127.12.126.66:37873)
0109 21:07:23.842878 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:23.842894 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:23.842979 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:23.843253 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:23.843257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:23.843275 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:23.843516 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:23.843522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.844263 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.844266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:23.844947 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:23.844955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.845057 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.845078 (+ 21us) client_negotiation.cc:769] Sending connection context
0109 21:07:23.845130 (+ 52us) client_negotiation.cc:240] Negotiation successful
0109 21:07:23.845185 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20250109 21:07:23.845767 27572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.842843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:37873 (local address 127.12.126.67:40177)
0109 21:07:23.842964 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:23.842968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:23.842980 (+ 12us) server_negotiation.cc:407] Connection header received
0109 21:07:23.843116 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:23.843119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:23.843159 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:23.843232 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:23.843620 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.844124 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.845066 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.845517 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.845573 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:23.845613 (+ 40us) server_negotiation.cc:299] Negotiation successful
0109 21:07:23.845668 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250109 21:07:23.845930 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.842722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:34057 (local address 127.12.126.66:53349)
0109 21:07:23.843144 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:23.843154 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:23.843234 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:23.843522 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:23.843524 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:23.843540 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:23.843762 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:23.843767 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.844568 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.844573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:23.845140 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:23.845148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.845255 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.845273 (+ 18us) client_negotiation.cc:769] Sending connection context
0109 21:07:23.845333 (+ 60us) client_negotiation.cc:240] Negotiation successful
0109 21:07:23.845682 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":130,"thread_start_us":95,"threads_started":1}
I20250109 21:07:23.845992 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.842864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:53349 (local address 127.12.126.65:34057)
0109 21:07:23.843004 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:23.843009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:23.843226 (+ 217us) server_negotiation.cc:407] Connection header received
0109 21:07:23.843368 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:23.843371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:23.843443 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:23.843507 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:23.843868 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.844451 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.845264 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.845733 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.845767 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:23.845817 (+ 50us) server_negotiation.cc:299] Negotiation successful
0109 21:07:23.845879 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250109 21:07:23.846650 27507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134dac1af42f4c939db1688ba295eb1f" candidate_uuid: "fd658876396e4ee6a9097fe40fbb39af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1ea46aded234e3bbecd1e6237446365" is_pre_election: true
I20250109 21:07:23.846652 27245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134dac1af42f4c939db1688ba295eb1f" candidate_uuid: "fd658876396e4ee6a9097fe40fbb39af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59d4d98f404eb6ab752bd586912096" is_pre_election: true
I20250109 21:07:23.846905 27507 raft_consensus.cc:2463] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd658876396e4ee6a9097fe40fbb39af in term 0.
I20250109 21:07:23.846899 27245 raft_consensus.cc:2463] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd658876396e4ee6a9097fe40fbb39af in term 0.
I20250109 21:07:23.847318 27317 leader_election.cc:304] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c1ea46aded234e3bbecd1e6237446365, cd59d4d98f404eb6ab752bd586912096, fd658876396e4ee6a9097fe40fbb39af; no voters:
I20250109 21:07:23.847534 27579 raft_consensus.cc:2798] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:07:23.847599 27579 raft_consensus.cc:491] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:07:23.847628 27579 raft_consensus.cc:3054] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:23.848546 27579 raft_consensus.cc:513] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } } peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } }
I20250109 21:07:23.848841 27579 leader_election.cc:290] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [CANDIDATE]: Term 1 election: Requested vote from peers c1ea46aded234e3bbecd1e6237446365 (127.12.126.67:40177), cd59d4d98f404eb6ab752bd586912096 (127.12.126.65:34057)
I20250109 21:07:23.849139 27507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134dac1af42f4c939db1688ba295eb1f" candidate_uuid: "fd658876396e4ee6a9097fe40fbb39af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1ea46aded234e3bbecd1e6237446365"
I20250109 21:07:23.849140 27245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134dac1af42f4c939db1688ba295eb1f" candidate_uuid: "fd658876396e4ee6a9097fe40fbb39af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59d4d98f404eb6ab752bd586912096"
I20250109 21:07:23.849231 27507 raft_consensus.cc:3054] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:23.849256 27245 raft_consensus.cc:3054] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:23.850104 27507 raft_consensus.cc:2463] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd658876396e4ee6a9097fe40fbb39af in term 1.
I20250109 21:07:23.850135 27245 raft_consensus.cc:2463] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd658876396e4ee6a9097fe40fbb39af in term 1.
I20250109 21:07:23.850376 27316 leader_election.cc:304] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1ea46aded234e3bbecd1e6237446365, fd658876396e4ee6a9097fe40fbb39af; no voters:
I20250109 21:07:23.850557 27579 raft_consensus.cc:2798] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:07:23.850759 27579 raft_consensus.cc:695] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [term 1 LEADER]: Becoming Leader. State: Replica: fd658876396e4ee6a9097fe40fbb39af, State: Running, Role: LEADER
I20250109 21:07:23.851058 27579 consensus_queue.cc:237] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } } peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } }
I20250109 21:07:23.853595 27103 catalog_manager.cc:5526] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af reported cstate change: term changed from 0 to 1, leader changed from <none> to fd658876396e4ee6a9097fe40fbb39af (127.12.126.66). New cstate: current_term: 1 leader_uuid: "fd658876396e4ee6a9097fe40fbb39af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } health_report { overall_health: UNKNOWN } } }
I20250109 21:07:23.865247 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:23.862878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.12.126.66:38203)
0109 21:07:23.863005 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:23.863008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:23.863019 (+ 11us) server_negotiation.cc:407] Connection header received
0109 21:07:23.863079 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:23.863081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:23.863124 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:23.863209 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:07:23.863459 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.863902 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:23.864427 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:23.864547 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:23.864616 (+ 69us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:07:23.864968 (+ 352us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:07:23.865082 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:23.865110 (+ 28us) server_negotiation.cc:299] Negotiation successful
0109 21:07:23.865149 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20250109 21:07:23.870051 27245 raft_consensus.cc:1270] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Refusing update from remote peer fd658876396e4ee6a9097fe40fbb39af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:07:23.870059 27507 raft_consensus.cc:1270] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365 [term 1 FOLLOWER]: Refusing update from remote peer fd658876396e4ee6a9097fe40fbb39af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:07:23.870628 27579 consensus_queue.cc:1035] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:07:23.870837 27581 consensus_queue.cc:1035] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:07:23.885360 27103 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33694:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:07:23.885511 27103 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33694:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:07:23.887672 27103 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f: Sending DeleteTablet for 3 replicas of tablet 134dac1af42f4c939db1688ba295eb1f
I20250109 21:07:23.888207 27363 tablet_service.cc:1514] Processing DeleteTablet for tablet 134dac1af42f4c939db1688ba295eb1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:07:23 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:51416
I20250109 21:07:23.888257 27495 tablet_service.cc:1514] Processing DeleteTablet for tablet 134dac1af42f4c939db1688ba295eb1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:07:23 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:36946
I20250109 21:07:23.888331 27223 tablet_service.cc:1514] Processing DeleteTablet for tablet 134dac1af42f4c939db1688ba295eb1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:07:23 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:41782
I20250109 21:07:23.888478 27300 tablet_replica.cc:331] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096: stopping tablet replica
I20250109 21:07:23.888589 27300 raft_consensus.cc:2238] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:07:23.888698 27300 raft_consensus.cc:2267] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:07:23.888778 27594 tablet_replica.cc:331] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af: stopping tablet replica
I20250109 21:07:23.889007 27594 raft_consensus.cc:2238] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:07:23.889021 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 27141
I20250109 21:07:23.889384 27594 raft_consensus.cc:2267] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:07:23.889436 27300 ts_tablet_manager.cc:1905] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:07:23.890242 27595 tablet_replica.cc:331] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365: stopping tablet replica
I20250109 21:07:23.890468 27595 raft_consensus.cc:2238] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:07:23.890707 27595 raft_consensus.cc:2267] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:07:23.892215 27594 ts_tablet_manager.cc:1905] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:07:23.894726 27594 ts_tablet_manager.cc:1918] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:07:23.894836 27594 log.cc:1198] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/134dac1af42f4c939db1688ba295eb1f
I20250109 21:07:23.895179 27594 ts_tablet_manager.cc:1939] T 134dac1af42f4c939db1688ba295eb1f P fd658876396e4ee6a9097fe40fbb39af: Deleting consensus metadata
I20250109 21:07:23.895970 27595 ts_tablet_manager.cc:1905] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:07:23.896173 27090 catalog_manager.cc:4872] TS fd658876396e4ee6a9097fe40fbb39af (127.12.126.66:38203): tablet 134dac1af42f4c939db1688ba295eb1f (table test-table [id=2482343db42149a89dc2349d345dbbf3]) successfully deleted
W20250109 21:07:23.896253 27088 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.126.65:34057 (error 108)
W20250109 21:07:23.897197 27297 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:07:23.896876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:34057 (local address 127.0.0.1:59728)
0109 21:07:23.897036 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:23.897100 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.65:34057: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":86}
I20250109 21:07:23.897255 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 27296
W20250109 21:07:23.897900 27088 catalog_manager.cc:4615] TS cd59d4d98f404eb6ab752bd586912096 (127.12.126.65:34057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 134dac1af42f4c939db1688ba295eb1f: Network error: Client connection negotiation failed: client connection to 127.12.126.65:34057: connect: Connection reset by peer (error 104)
I20250109 21:07:23.900015 27595 ts_tablet_manager.cc:1918] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:07:23.900086 27595 log.cc:1198] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/134dac1af42f4c939db1688ba295eb1f
I20250109 21:07:23.900381 27595 ts_tablet_manager.cc:1939] T 134dac1af42f4c939db1688ba295eb1f P c1ea46aded234e3bbecd1e6237446365: Deleting consensus metadata
I20250109 21:07:23.901288 27087 catalog_manager.cc:4872] TS c1ea46aded234e3bbecd1e6237446365 (127.12.126.67:40177): tablet 134dac1af42f4c939db1688ba295eb1f (table test-table [id=2482343db42149a89dc2349d345dbbf3]) successfully deleted
I20250109 21:07:23.903432 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 27432
I20250109 21:07:23.908893 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 27070
I20250109 21:07:23.915205 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:33113
--webserver_interface=127.12.126.126
--webserver_port=46511
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:33113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:07:24.013428 27597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:24.013654 27597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:24.013731 27597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:24.016999 27597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:07:24.017076 27597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:24.017112 27597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:24.017151 27597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:07:24.017189 27597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:07:24.021374 27597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:33113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:33113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=46511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:24.022330 27597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:24.023156 27597 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:24.029548 27605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:24.030282 27602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:24.031054 27597 server_base.cc:1034] running on GCE node
W20250109 21:07:24.031808 27603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:24.032228 27597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:24.033001 27597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:24.034168 27597 hybrid_clock.cc:648] HybridClock initialized: now 1736456844034154 us; error 24 us; skew 500 ppm
Jan 09 21:07:24 dist-test-slave-g0zb krb5kdc[26524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456844, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:24.036728 27597 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:07:24.037551 27597 webserver.cc:458] Webserver started at http://127.12.126.126:46511/ using document root <none> and password file <none>
I20250109 21:07:24.038044 27597 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:24.038095 27597 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:24.041290 27597 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:07:24.043010 27612 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:24.043941 27597 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:07:24.044051 27597 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "72863e5fa41f5e8b9c842c8f307bad38"
server_key_iv: "f06545f6132740a1c757654e97bd4ae3"
server_key_version: "encryptionkey@0"
I20250109 21:07:24.044417 27597 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:24.063627 27597 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:24.065904 27597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:24.066062 27597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:24.072162 27597 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:33113
I20250109 21:07:24.072384 27664 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:33113 every 8 connection(s)
I20250109 21:07:24.073104 27597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:07:24.076614 27665 sys_catalog.cc:263] Verifying existing consensus state
I20250109 21:07:24.078020 27665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f: Bootstrap starting.
I20250109 21:07:24.079993 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 27597
I20250109 21:07:24.080387 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:34057
--local_ip_for_outbound_sockets=127.12.126.65
--tserver_master_addrs=127.12.126.126:33113
--webserver_port=35675
--webserver_interface=127.12.126.65
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:07:24.093961 27665 log.cc:826] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:24.103986 27665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:07:24.104334 27665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f: Bootstrap complete.
I20250109 21:07:24.107698 27665 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33113 } }
I20250109 21:07:24.107986 27665 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0e9f0299b74ce2ba7c731714c2cf6f, State: Initialized, Role: FOLLOWER
I20250109 21:07:24.108436 27665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33113 } }
I20250109 21:07:24.108566 27665 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:07:24.108632 27665 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:07:24.108732 27665 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 2 FOLLOWER]: Advancing to term 3
I20250109 21:07:24.110158 27665 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33113 } }
I20250109 21:07:24.110445 27665 leader_election.cc:304] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f0e9f0299b74ce2ba7c731714c2cf6f; no voters:
I20250109 21:07:24.110716 27665 leader_election.cc:290] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [CANDIDATE]: Term 3 election: Requested vote from peers
I20250109 21:07:24.110829 27669 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 3 FOLLOWER]: Leader election won for term 3
I20250109 21:07:24.111011 27669 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [term 3 LEADER]: Becoming Leader. State: Replica: 8f0e9f0299b74ce2ba7c731714c2cf6f, State: Running, Role: LEADER
I20250109 21:07:24.111263 27669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33113 } }
I20250109 21:07:24.111752 27665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:07:24.112082 27669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f0e9f0299b74ce2ba7c731714c2cf6f. Latest consensus state: current_term: 3 leader_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33113 } } }
I20250109 21:07:24.112063 27670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e9f0299b74ce2ba7c731714c2cf6f" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 33113 } } }
I20250109 21:07:24.112208 27669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [sys.catalog]: This master's current role is: LEADER
I20250109 21:07:24.112212 27670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0e9f0299b74ce2ba7c731714c2cf6f [sys.catalog]: This master's current role is: LEADER
I20250109 21:07:24.112650 27673 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:07:24.117213 27673 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:07:24.117857 27673 catalog_manager.cc:1260] Loaded cluster ID: ee0104f07acb45af8684c5ff9af5d1c7
I20250109 21:07:24.117909 27673 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
F20250109 21:07:24.118597 27673 catalog_manager.cc:1467] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbe68058dcd google::LogMessage::Fail() at ??:0
@ 0x7fbe6805cb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fbe680587cc google::LogMessage::Flush() at ??:0
@ 0x7fbe68059f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbe6cf6c4b2 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fbe6cf6d34a kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbe6cf6984b _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fbe6cfb0dd9 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5643fd32bbbe std::function<>::operator()() at ??:0
@ 0x7fbe694c3a48 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbe694c4331 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fbe694c5c7d _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5643fd32bbbe std::function<>::operator()() at ??:0
@ 0x7fbe694b3495 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbe69cfb6db start_thread at ??:0
@ 0x7fbe6720f71f clone at ??:0
W20250109 21:07:24.205546 27667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:24.205749 27667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:24.205801 27667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:24.208817 27667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:24.208868 27667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:24.208971 27667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:07:24.212852 27667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:34057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=35675
--tserver_master_addrs=127.12.126.126:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:24.213755 27667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:24.214540 27667 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:24.221905 27707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:24.222606 27706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:24.223172 27709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:24.224303 27667 server_base.cc:1034] running on GCE node
I20250109 21:07:24.224617 27667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:24.225101 27667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:24.226260 27667 hybrid_clock.cc:648] HybridClock initialized: now 1736456844226248 us; error 30 us; skew 500 ppm
Jan 09 21:07:24 dist-test-slave-g0zb krb5kdc[26524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456844, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:24.228654 27667 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:07:24.229449 27667 webserver.cc:458] Webserver started at http://127.12.126.65:35675/ using document root <none> and password file <none>
I20250109 21:07:24.229964 27667 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:24.230016 27667 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:24.233297 27667 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20250109 21:07:24.235143 27716 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:24.236121 27667 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250109 21:07:24.236213 27667 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "cd59d4d98f404eb6ab752bd586912096"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "e3749b2e75136e80788ab3049b38d2ef"
server_key_iv: "6e7ca46d61ba11a4bfc6aa76fc8d3865"
server_key_version: "encryptionkey@0"
I20250109 21:07:24.236605 27667 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:24.251587 27667 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:24.254297 27667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:24.254447 27667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:24.254961 27667 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:07:24.255911 27723 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250109 21:07:24.259295 27667 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250109 21:07:24.259361 27667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20250109 21:07:24.259407 27667 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250109 21:07:24.260807 27667 ts_tablet_manager.cc:610] Registered 1 tablets
I20250109 21:07:24.260855 27667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250109 21:07:24.261090 27723 tablet_bootstrap.cc:492] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096: Bootstrap starting.
I20250109 21:07:24.266613 27723 log.cc:826] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:24.270671 27667 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:34057
I20250109 21:07:24.270924 27831 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:34057 every 8 connection(s)
I20250109 21:07:24.271642 27667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:07:24.274585 27723 tablet_bootstrap.cc:492] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:07:24.274971 27723 tablet_bootstrap.cc:492] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096: Bootstrap complete.
I20250109 21:07:24.275239 27723 ts_tablet_manager.cc:1397] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.003s
I20250109 21:07:24.275972 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 27667
I20250109 21:07:24.276455 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:38203
--local_ip_for_outbound_sockets=127.12.126.66
--tserver_master_addrs=127.12.126.126:33113
--webserver_port=37509
--webserver_interface=127.12.126.66
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:07:24.277127 27723 raft_consensus.cc:357] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } } peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } }
I20250109 21:07:24.277415 27723 raft_consensus.cc:738] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59d4d98f404eb6ab752bd586912096, State: Initialized, Role: FOLLOWER
I20250109 21:07:24.277848 27723 consensus_queue.cc:260] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } } peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } }
I20250109 21:07:24.278741 27723 ts_tablet_manager.cc:1428] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
W20250109 21:07:24.335860 27834 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:07:24.273486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:34791)
0109 21:07:24.274000 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:24.274055 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:24.274875 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:24.335165 (+ 60290us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":114,"threads_started":1}
W20250109 21:07:24.336560 27832 heartbeater.cc:643] Failed to heartbeat to 127.12.126.126:33113 (0 consecutive failures): Network error: Failed to ping master at 127.12.126.126:33113: Client connection negotiation failed: client connection to 127.12.126.126:33113: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250109 21:07:24.337280 27834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:24.337011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:53009)
0109 21:07:24.337142 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:24.337211 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20250109 21:07:24.337872 27834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:24.337668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:34659)
0109 21:07:24.337776 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:24.337818 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20250109 21:07:24.338009 27832 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250109 21:07:24.394984 27835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:24.395188 27835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:24.395242 27835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:24.398216 27835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:24.398267 27835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:24.398370 27835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:07:24.402312 27835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:38203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=37509
--tserver_master_addrs=127.12.126.126:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:24.403191 27835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:24.403939 27835 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:24.411440 27844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:24.412161 27841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:24.412724 27842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:24.413519 27835 server_base.cc:1034] running on GCE node
I20250109 21:07:24.413821 27835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:24.414297 27835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:24.415442 27835 hybrid_clock.cc:648] HybridClock initialized: now 1736456844415432 us; error 31 us; skew 500 ppm
Jan 09 21:07:24 dist-test-slave-g0zb krb5kdc[26524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456844, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:24.417733 27835 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:07:24.418471 27835 webserver.cc:458] Webserver started at http://127.12.126.66:37509/ using document root <none> and password file <none>
I20250109 21:07:24.418994 27835 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:24.419073 27835 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:24.422178 27835 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:07:24.423877 27851 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:24.424780 27835 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:07:24.424880 27835 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "fd658876396e4ee6a9097fe40fbb39af"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "cdb29647d56ea3a6c64565f7c376763d"
server_key_iv: "f1aa231fa9758b0f7b6e050e754f9bc4"
server_key_version: "encryptionkey@0"
I20250109 21:07:24.425243 27835 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:24.439847 27835 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:24.442557 27835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:24.442708 27835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:24.443209 27835 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:07:24.443890 27835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:07:24.443938 27835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:24.443990 27835 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:07:24.444020 27835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:24.452077 27835 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:38203
I20250109 21:07:24.452107 27964 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:38203 every 8 connection(s)
I20250109 21:07:24.452961 27835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:07:24.453544 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 27835
I20250109 21:07:24.453907 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:40177
--local_ip_for_outbound_sockets=127.12.126.67
--tserver_master_addrs=127.12.126.126:33113
--webserver_port=35379
--webserver_interface=127.12.126.67
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:07:24.456125 27968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:24.455077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:42133)
0109 21:07:24.455500 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:24.455582 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":125,"threads_started":1}
W20250109 21:07:24.456480 27965 heartbeater.cc:643] Failed to heartbeat to 127.12.126.126:33113 (0 consecutive failures): Network error: Failed to ping master at 127.12.126.126:33113: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
I20250109 21:07:24.457031 27968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:24.456761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:33755)
0109 21:07:24.456901 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:24.456956 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20250109 21:07:24.457680 27968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:24.457478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:46031)
0109 21:07:24.457594 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:24.457631 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20250109 21:07:24.457844 27965 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250109 21:07:24.551934 27967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:24.552131 27967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:24.552196 27967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:24.555124 27967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:24.555171 27967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:24.555270 27967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:07:24.558907 27967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:40177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=35379
--tserver_master_addrs=127.12.126.126:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:24.559799 27967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:24.560567 27967 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:24.567759 27976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:24.568437 27974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:24.569237 27973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:24.570111 27967 server_base.cc:1034] running on GCE node
I20250109 21:07:24.570415 27967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:24.570917 27967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:24.572057 27967 hybrid_clock.cc:648] HybridClock initialized: now 1736456844572044 us; error 25 us; skew 500 ppm
Jan 09 21:07:24 dist-test-slave-g0zb krb5kdc[26524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456844, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:24.574429 27967 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:07:24.575223 27967 webserver.cc:458] Webserver started at http://127.12.126.67:35379/ using document root <none> and password file <none>
I20250109 21:07:24.575680 27967 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:24.575726 27967 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:24.578748 27967 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:07:24.580478 27983 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:24.581357 27967 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:07:24.581439 27967 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "c1ea46aded234e3bbecd1e6237446365"
format_stamp: "Formatted at 2025-01-09 21:07:22 on dist-test-slave-g0zb"
server_key: "484d5a0687f6a34f806aa85bd0c883cb"
server_key_iv: "a4b8f9ca5b38c31294b001e4cadb195c"
server_key_version: "encryptionkey@0"
I20250109 21:07:24.581770 27967 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:24.601217 27967 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:24.603519 27967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:24.603665 27967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:24.604108 27967 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:07:24.604776 27967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:07:24.604820 27967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:24.604861 27967 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:07:24.604890 27967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:24.613071 27967 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:40177
I20250109 21:07:24.613114 28096 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:40177 every 8 connection(s)
I20250109 21:07:24.614001 27967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:07:24.616967 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:24.616040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:49217)
0109 21:07:24.616401 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:24.616483 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":87,"threads_started":1}
W20250109 21:07:24.617367 28097 heartbeater.cc:643] Failed to heartbeat to 127.12.126.126:33113 (0 consecutive failures): Network error: Failed to ping master at 127.12.126.126:33113: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
I20250109 21:07:24.617862 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:24.617670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:33925)
0109 21:07:24.617773 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:24.617814 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20250109 21:07:24.618377 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:24.618193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:35973)
0109 21:07:24.618297 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:24.618336 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20250109 21:07:24.618522 28097 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20250109 21:07:24.619762 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 27967
I20250109 21:07:25.339004 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:25.338589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:49605)
0109 21:07:25.338859 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:25.338907 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20250109 21:07:25.458765 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:25.458314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:40695)
0109 21:07:25.458623 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:25.458673 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":78,"threads_started":1}
I20250109 21:07:25.575300 28103 raft_consensus.cc:491] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:07:25.575501 28103 raft_consensus.cc:513] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } } peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } }
I20250109 21:07:25.576545 28103 leader_election.cc:290] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd658876396e4ee6a9097fe40fbb39af (127.12.126.66:38203), c1ea46aded234e3bbecd1e6237446365 (127.12.126.67:40177)
Jan 09 21:07:25 dist-test-slave-g0zb krb5kdc[26524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456844, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.67@KRBTEST.COM
Jan 09 21:07:25 dist-test-slave-g0zb krb5kdc[26524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456844, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.66@KRBTEST.COM
I20250109 21:07:25.588001 28106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:25.576920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:54749 (local address 127.12.126.67:40177)
0109 21:07:25.577248 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:25.577258 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:25.577291 (+ 33us) server_negotiation.cc:407] Connection header received
0109 21:07:25.577837 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:25.577842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:25.578135 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:25.578389 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:25.579765 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:25.580541 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:25.581181 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:25.581899 (+ 718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:25.583886 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:25.583938 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:25.583947 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:25.583976 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:25.585807 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:25.586339 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:25.586343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:25.586346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:25.586410 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:25.586644 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:25.586647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:25.586649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:25.586996 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:25.587118 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:25.587601 (+ 483us) server_negotiation.cc:299] Negotiation successful
0109 21:07:25.587825 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":77,"threads_started":1}
I20250109 21:07:25.588008 28105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:25.576921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:53777 (local address 127.12.126.66:38203)
0109 21:07:25.577299 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:25.577308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:25.577337 (+ 29us) server_negotiation.cc:407] Connection header received
0109 21:07:25.577857 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:25.577861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:25.578127 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:25.578407 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:25.579772 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:25.580537 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:25.581134 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:25.581383 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:25.584096 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:25.584125 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:25.584132 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:25.584161 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:25.586026 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:25.586414 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:25.586419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:25.586425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:25.586483 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:25.586676 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:25.586679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:25.586680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:25.587003 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:25.587118 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:25.587597 (+ 479us) server_negotiation.cc:299] Negotiation successful
0109 21:07:25.587823 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":72,"threads_started":1}
I20250109 21:07:25.588001 28104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:25.576777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:40177 (local address 127.12.126.65:54749)
0109 21:07:25.577109 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:25.577123 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:25.577193 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:25.578861 (+ 1668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:25.578866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:25.579127 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:25.579602 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:25.579625 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:25.580685 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:25.580688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:25.581066 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:25.581074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:25.581183 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:25.581836 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:25.581871 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:25.583720 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:25.585922 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:25.585927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:25.585937 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:25.586211 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:25.586493 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:25.586496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:25.586497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:25.586567 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:25.587100 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:25.587105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:25.587284 (+ 179us) client_negotiation.cc:769] Sending connection context
0109 21:07:25.587474 (+ 190us) client_negotiation.cc:240] Negotiation successful
0109 21:07:25.587719 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"spinlock_wait_cycles":21632,"thread_start_us":66,"threads_started":1}
I20250109 21:07:25.588011 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:25.576722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:38203 (local address 127.12.126.65:53777)
0109 21:07:25.576907 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:25.576923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:25.577023 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:25.578477 (+ 1454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:25.578484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:25.579086 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:25.579613 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:25.579623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:25.580696 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:25.580699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:25.581035 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:25.581041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:25.581181 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:25.581901 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:25.581915 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:25.583969 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:25.586121 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:25.586124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:25.586126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:25.586305 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:25.586559 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:25.586563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:25.586565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:25.586605 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:25.587102 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:25.587109 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:25.587284 (+ 175us) client_negotiation.cc:769] Sending connection context
0109 21:07:25.587474 (+ 190us) client_negotiation.cc:240] Negotiation successful
0109 21:07:25.587824 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":126,"spinlock_wait_cycles":3584}
W20250109 21:07:25.590164 27717 leader_election.cc:343] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1ea46aded234e3bbecd1e6237446365 (127.12.126.67:40177): Not found: Tablet not found: 134dac1af42f4c939db1688ba295eb1f
W20250109 21:07:25.590325 27720 leader_election.cc:343] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd658876396e4ee6a9097fe40fbb39af (127.12.126.66:38203): Not found: Tablet not found: 134dac1af42f4c939db1688ba295eb1f
I20250109 21:07:25.590397 27720 leader_election.cc:304] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd59d4d98f404eb6ab752bd586912096; no voters: c1ea46aded234e3bbecd1e6237446365, fd658876396e4ee6a9097fe40fbb39af
I20250109 21:07:25.590595 28103 raft_consensus.cc:2743] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 21:07:25.619320 28107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:25.618983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:52287)
0109 21:07:25.619206 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:25.619250 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":80,"threads_started":1}
I20250109 21:07:26.340073 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:26.339690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:60871)
0109 21:07:26.339927 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:26.339976 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":73,"threads_started":1}
I20250109 21:07:26.459869 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:26.459502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:50963)
0109 21:07:26.459737 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:26.459784 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":79,"threads_started":1}
I20250109 21:07:26.620375 28110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:26.619990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:50129)
0109 21:07:26.620217 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:26.620269 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20250109 21:07:27.341323 28111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:27.340878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:47811)
0109 21:07:27.341162 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:27.341224 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":80,"threads_started":1}
I20250109 21:07:27.461153 28112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:27.460657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:60295)
0109 21:07:27.460948 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:27.461046 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":76,"threads_started":1}
I20250109 21:07:27.490098 28113 raft_consensus.cc:491] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:07:27.490235 28113 raft_consensus.cc:513] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } } peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } }
I20250109 21:07:27.490630 28113 leader_election.cc:290] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd658876396e4ee6a9097fe40fbb39af (127.12.126.66:38203), c1ea46aded234e3bbecd1e6237446365 (127.12.126.67:40177)
W20250109 21:07:27.491462 27717 leader_election.cc:343] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1ea46aded234e3bbecd1e6237446365 (127.12.126.67:40177): Not found: Tablet not found: 134dac1af42f4c939db1688ba295eb1f
W20250109 21:07:27.491549 27720 leader_election.cc:343] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd658876396e4ee6a9097fe40fbb39af (127.12.126.66:38203): Not found: Tablet not found: 134dac1af42f4c939db1688ba295eb1f
I20250109 21:07:27.491648 27720 leader_election.cc:304] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd59d4d98f404eb6ab752bd586912096; no voters: c1ea46aded234e3bbecd1e6237446365, fd658876396e4ee6a9097fe40fbb39af
I20250109 21:07:27.491824 28113 raft_consensus.cc:2743] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 21:07:27.621470 28114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:27.621059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:50265)
0109 21:07:27.621326 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:27.621385 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
I20250109 21:07:28.342535 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:28.342052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:48463)
0109 21:07:28.342327 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:28.342391 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":97,"threads_started":1}
I20250109 21:07:28.462332 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:28.461978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:45401)
0109 21:07:28.462193 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:28.462241 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":69,"threads_started":1}
I20250109 21:07:28.622452 28117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:28.622066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:35929)
0109 21:07:28.622315 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:28.622367 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
I20250109 21:07:29.343746 28118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:29.343345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:39187)
0109 21:07:29.343600 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:29.343650 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":72,"threads_started":1}
I20250109 21:07:29.463346 28119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:29.462980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:34439)
0109 21:07:29.463208 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:29.463259 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
I20250109 21:07:29.623494 28120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:29.623115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:52755)
0109 21:07:29.623352 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:29.623404 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20250109 21:07:30.344826 28121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:30.344432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:39457)
0109 21:07:30.344682 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:30.344729 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
I20250109 21:07:30.464344 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:30.463958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:35569)
0109 21:07:30.464211 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:30.464261 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20250109 21:07:30.624543 28123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:30.624147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:60263)
0109 21:07:30.624407 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:30.624458 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
I20250109 21:07:30.707855 28124 raft_consensus.cc:491] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:07:30.707983 28124 raft_consensus.cc:513] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } } peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } }
I20250109 21:07:30.708384 28124 leader_election.cc:290] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd658876396e4ee6a9097fe40fbb39af (127.12.126.66:38203), c1ea46aded234e3bbecd1e6237446365 (127.12.126.67:40177)
W20250109 21:07:30.709064 27720 leader_election.cc:343] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd658876396e4ee6a9097fe40fbb39af (127.12.126.66:38203): Not found: Tablet not found: 134dac1af42f4c939db1688ba295eb1f
W20250109 21:07:30.709157 27717 leader_election.cc:343] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1ea46aded234e3bbecd1e6237446365 (127.12.126.67:40177): Not found: Tablet not found: 134dac1af42f4c939db1688ba295eb1f
I20250109 21:07:30.709244 27717 leader_election.cc:304] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd59d4d98f404eb6ab752bd586912096; no voters: c1ea46aded234e3bbecd1e6237446365, fd658876396e4ee6a9097fe40fbb39af
I20250109 21:07:30.709385 28124 raft_consensus.cc:2743] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 21:07:31.345984 28125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:31.345580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:52481)
0109 21:07:31.345823 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:31.345875 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":86,"threads_started":1}
I20250109 21:07:31.465497 28126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:31.465083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:34177)
0109 21:07:31.465349 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:31.465405 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":72,"threads_started":1}
I20250109 21:07:31.625730 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:31.625289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:58991)
0109 21:07:31.625579 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:31.625631 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":92,"threads_started":1}
I20250109 21:07:32.347084 28128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:32.346681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:49247)
0109 21:07:32.346937 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:32.346987 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":74,"threads_started":1}
I20250109 21:07:32.466490 28129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:32.466132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:35891)
0109 21:07:32.466354 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:32.466402 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":73,"threads_started":1}
I20250109 21:07:32.626827 28130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:32.626436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:43995)
0109 21:07:32.626663 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:32.626710 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
I20250109 21:07:32.825106 28131 raft_consensus.cc:491] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:07:32.825224 28131 raft_consensus.cc:513] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } } peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } }
I20250109 21:07:32.825572 28131 leader_election.cc:290] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd658876396e4ee6a9097fe40fbb39af (127.12.126.66:38203), c1ea46aded234e3bbecd1e6237446365 (127.12.126.67:40177)
W20250109 21:07:32.826231 27720 leader_election.cc:343] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd658876396e4ee6a9097fe40fbb39af (127.12.126.66:38203): Not found: Tablet not found: 134dac1af42f4c939db1688ba295eb1f
W20250109 21:07:32.826308 27717 leader_election.cc:343] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1ea46aded234e3bbecd1e6237446365 (127.12.126.67:40177): Not found: Tablet not found: 134dac1af42f4c939db1688ba295eb1f
I20250109 21:07:32.826391 27717 leader_election.cc:304] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd59d4d98f404eb6ab752bd586912096; no voters: c1ea46aded234e3bbecd1e6237446365, fd658876396e4ee6a9097fe40fbb39af
I20250109 21:07:32.826517 28131 raft_consensus.cc:2743] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 21:07:33.348143 28132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:33.347727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:35553)
0109 21:07:33.347996 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:33.348045 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20250109 21:07:33.467578 28133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:33.467137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:54711)
0109 21:07:33.467431 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:33.467482 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
I20250109 21:07:33.627852 28134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:33.627488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:33375)
0109 21:07:33.627718 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:33.627766 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
I20250109 21:07:34.349341 28135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:34.348964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:58487)
0109 21:07:34.349199 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:34.349247 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":79,"threads_started":1}
I20250109 21:07:34.468552 28136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:34.468172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:48437)
0109 21:07:34.468412 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:34.468464 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":75,"threads_started":1}
I20250109 21:07:34.628990 28137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:34.628579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:54339)
0109 21:07:34.628844 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:34.628903 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
I20250109 21:07:35.350503 28138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:35.350052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:41487)
0109 21:07:35.350346 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:35.350401 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":89,"threads_started":1}
I20250109 21:07:35.469743 28139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:35.469351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:34051)
0109 21:07:35.469595 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:35.469650 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":80,"threads_started":1}
I20250109 21:07:35.630106 28140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:35.629730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:42735)
0109 21:07:35.629956 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:35.630017 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20250109 21:07:36.351716 28141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:36.351291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:36591)
0109 21:07:36.351549 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:36.351605 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":91,"threads_started":1}
I20250109 21:07:36.471021 28142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:36.470528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:46533)
0109 21:07:36.470839 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:36.470898 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":78,"threads_started":1}
I20250109 21:07:36.631243 28143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:36.630799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:53677)
0109 21:07:36.631078 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:36.631138 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":78,"threads_started":1}
I20250109 21:07:37.352823 28144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:37.352453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:57345)
0109 21:07:37.352682 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:37.352729 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
I20250109 21:07:37.472113 28145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:37.471723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:33979)
0109 21:07:37.471972 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:37.472021 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
I20250109 21:07:37.632333 28146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:37.631880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:46131)
0109 21:07:37.632171 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:37.632227 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":90,"threads_started":1}
I20250109 21:07:38.354168 28147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:38.353716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:56021)
0109 21:07:38.354014 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:38.354070 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":93,"threads_started":1}
I20250109 21:07:38.473271 28148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:38.472874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:35569)
0109 21:07:38.473116 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:38.473174 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
I20250109 21:07:38.633572 28149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:38.633186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:42925)
0109 21:07:38.633431 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:38.633482 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":78,"threads_started":1}
I20250109 21:07:38.911602 28150 raft_consensus.cc:491] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:07:38.911747 28150 raft_consensus.cc:513] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd658876396e4ee6a9097fe40fbb39af" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 38203 } } peers { permanent_uuid: "c1ea46aded234e3bbecd1e6237446365" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40177 } } peers { permanent_uuid: "cd59d4d98f404eb6ab752bd586912096" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34057 } }
I20250109 21:07:38.912197 28150 leader_election.cc:290] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd658876396e4ee6a9097fe40fbb39af (127.12.126.66:38203), c1ea46aded234e3bbecd1e6237446365 (127.12.126.67:40177)
W20250109 21:07:38.912919 27717 leader_election.cc:343] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1ea46aded234e3bbecd1e6237446365 (127.12.126.67:40177): Not found: Tablet not found: 134dac1af42f4c939db1688ba295eb1f
W20250109 21:07:38.913002 27720 leader_election.cc:343] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd658876396e4ee6a9097fe40fbb39af (127.12.126.66:38203): Not found: Tablet not found: 134dac1af42f4c939db1688ba295eb1f
I20250109 21:07:38.913094 27720 leader_election.cc:304] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd59d4d98f404eb6ab752bd586912096; no voters: c1ea46aded234e3bbecd1e6237446365, fd658876396e4ee6a9097fe40fbb39af
I20250109 21:07:38.913251 28150 raft_consensus.cc:2743] T 134dac1af42f4c939db1688ba295eb1f P cd59d4d98f404eb6ab752bd586912096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 21:07:39.355619 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:39.355175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.65:58161)
0109 21:07:39.355453 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:39.355507 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":71,"threads_started":1}
I20250109 21:07:39.474452 28152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:39.474048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.66:49183)
0109 21:07:39.474308 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:39.474362 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":71,"threads_started":1}
I20250109 21:07:39.625327 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 27667
I20250109 21:07:39.631162 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 27835
I20250109 21:07:39.634691 28153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:39.634347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:33113 (local address 127.12.126.67:38011)
0109 21:07:39.634565 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:39.634610 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":63,"threads_started":1}
I20250109 21:07:39.636577 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 27967
2025-01-09T21:07:39Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18701 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:07:39 dist-test-slave-g0zb krb5kdc[28156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:07:39 dist-test-slave-g0zb krb5kdc[28156](info): set up 2 sockets
Jan 09 21:07:39 dist-test-slave-g0zb krb5kdc[28156](info): commencing operation
krb5kdc: starting...
W20250109 21:07:40.643630 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:07:40 dist-test-slave-g0zb krb5kdc[28156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:07:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:07:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:07:40.755430 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:35275
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:35275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:07:40.860087 28169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:40.860337 28169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:40.860388 28169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:40.863514 28169 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20250109 21:07:40.863569 28169 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20250109 21:07:40.863584 28169 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20250109 21:07:40.863600 28169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:07:40.863618 28169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:40.863639 28169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:40.863656 28169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:07:40.863672 28169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:07:40.867826 28169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:35275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:35275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:40.868885 28169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:40.869697 28169 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:40.876539 28175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:40.877226 28174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:40.877774 28177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:40.878448 28169 server_base.cc:1034] running on GCE node
I20250109 21:07:40.878984 28169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:40.879921 28169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:40.881088 28169 hybrid_clock.cc:648] HybridClock initialized: now 1736456860881074 us; error 25 us; skew 500 ppm
Jan 09 21:07:40 dist-test-slave-g0zb krb5kdc[28156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456860, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:40.884020 28169 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:07:40.884898 28169 webserver.cc:458] Webserver started at http://127.12.126.126:46205/ using document root <none> and password file <none>
I20250109 21:07:40.885511 28169 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:40.885567 28169 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:40.885746 28169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:07:40.887523 28169 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "cf5434de2d4448068da2c603fb583b04"
format_stamp: "Formatted at 2025-01-09 21:07:40 on dist-test-slave-g0zb"
server_key: "33320c6b3e8aa2962f9f4bfff1b9b7b3"
server_key_iv: "74abe7bf2c0fffc09886a847e60e63a3"
server_key_version: "encryptionkey@0"
I20250109 21:07:40.887980 28169 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "cf5434de2d4448068da2c603fb583b04"
format_stamp: "Formatted at 2025-01-09 21:07:40 on dist-test-slave-g0zb"
server_key: "33320c6b3e8aa2962f9f4bfff1b9b7b3"
server_key_iv: "74abe7bf2c0fffc09886a847e60e63a3"
server_key_version: "encryptionkey@0"
I20250109 21:07:40.891134 28169 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:07:40.893277 28184 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:40.894193 28169 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:07:40.894297 28169 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "cf5434de2d4448068da2c603fb583b04"
format_stamp: "Formatted at 2025-01-09 21:07:40 on dist-test-slave-g0zb"
server_key: "33320c6b3e8aa2962f9f4bfff1b9b7b3"
server_key_iv: "74abe7bf2c0fffc09886a847e60e63a3"
server_key_version: "encryptionkey@0"
I20250109 21:07:40.894376 28169 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:40.911501 28169 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:40.914186 28169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:40.914353 28169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:40.920763 28169 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:35275
I20250109 21:07:40.921010 28236 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:35275 every 8 connection(s)
I20250109 21:07:40.921741 28169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:07:40.924208 28237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:40.930897 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 28169
I20250109 21:07:40.931066 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:07:40.931210 28237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: Bootstrap starting.
I20250109 21:07:40.931386 12793 external_mini_cluster.cc:1453] Setting key 1918264114a088bc05b561d5db939d99
I20250109 21:07:40.934285 28237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: Neither blocks nor log segments found. Creating new log.
I20250109 21:07:40.935182 28237 log.cc:826] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:40.939076 28237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: No bootstrap required, opened a new log
Jan 09 21:07:40 dist-test-slave-g0zb krb5kdc[28156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:07:40.943739 28237 raft_consensus.cc:357] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5434de2d4448068da2c603fb583b04" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35275 } }
I20250109 21:07:40.944029 28237 raft_consensus.cc:383] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:40.944128 28237 raft_consensus.cc:738] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5434de2d4448068da2c603fb583b04, State: Initialized, Role: FOLLOWER
I20250109 21:07:40.944664 28237 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5434de2d4448068da2c603fb583b04" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35275 } }
I20250109 21:07:40.944788 28237 raft_consensus.cc:397] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:07:40.944849 28237 raft_consensus.cc:491] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:07:40.944954 28237 raft_consensus.cc:3054] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:40.945878 28237 raft_consensus.cc:513] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5434de2d4448068da2c603fb583b04" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35275 } }
I20250109 21:07:40.946270 28237 leader_election.cc:304] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf5434de2d4448068da2c603fb583b04; no voters:
I20250109 21:07:40.946588 28237 leader_election.cc:290] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:07:40.946718 28242 raft_consensus.cc:2798] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:07:40.946926 28242 raft_consensus.cc:695] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 1 LEADER]: Becoming Leader. State: Replica: cf5434de2d4448068da2c603fb583b04, State: Running, Role: LEADER
I20250109 21:07:40.947268 28242 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5434de2d4448068da2c603fb583b04" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35275 } }
I20250109 21:07:40.947803 28237 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:07:40.948465 28240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:40.933095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36186 (local address 127.12.126.126:35275)
0109 21:07:40.934539 (+ 1444us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:40.934554 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:40.934599 (+ 45us) server_negotiation.cc:407] Connection header received
0109 21:07:40.935417 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:40.935443 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:40.935830 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:40.936247 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:40.937465 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:40.938802 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:40.940132 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:40.940447 (+ 315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:40.943222 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:40.943251 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:40.943269 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:40.943318 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:40.945460 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:40.945962 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:40.945970 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:40.945976 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:40.946075 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:40.946348 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:40.946352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:40.946355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:40.946864 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:40.947072 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:40.947406 (+ 334us) server_negotiation.cc:299] Negotiation successful
0109 21:07:40.947676 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1281,"thread_start_us":126,"threads_started":1}
I20250109 21:07:40.948508 28242 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf5434de2d4448068da2c603fb583b04. Latest consensus state: current_term: 1 leader_uuid: "cf5434de2d4448068da2c603fb583b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5434de2d4448068da2c603fb583b04" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35275 } } }
I20250109 21:07:40.948621 28242 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [sys.catalog]: This master's current role is: LEADER
I20250109 21:07:40.950459 28243 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf5434de2d4448068da2c603fb583b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5434de2d4448068da2c603fb583b04" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35275 } } }
I20250109 21:07:40.950549 28243 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [sys.catalog]: This master's current role is: LEADER
W20250109 21:07:40.952225 28257 catalog_manager.cc:1559] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:07:40.952288 28257 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 21:07:40.952348 28249 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:07:40.952973 28249 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:07:40.958282 28249 catalog_manager.cc:1348] Generated new cluster ID: 2f406a50bda84df8bbf80dc3ab5f3401
I20250109 21:07:40.958351 28249 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:07:40.973570 28249 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:07:40.974484 28249 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:07:40.981431 28249 catalog_manager.cc:5899] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: Generated new TSK 0
I20250109 21:07:40.982097 28249 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
I20250109 21:07:41.007951 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35275
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:07:41.112682 28263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:41.112896 28263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:41.112960 28263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:41.115864 28263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:41.115919 28263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:41.116011 28263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:07:41.119910 28263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:41.120869 28263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:41.121711 28263 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:41.130003 28268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:41.130148 28269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:41.131018 28271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:41.131944 28263 server_base.cc:1034] running on GCE node
I20250109 21:07:41.132328 28263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:41.132884 28263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:41.134023 28263 hybrid_clock.cc:648] HybridClock initialized: now 1736456861134013 us; error 23 us; skew 500 ppm
Jan 09 21:07:41 dist-test-slave-g0zb krb5kdc[28156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456861, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:41.136529 28263 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:07:41.137408 28263 webserver.cc:458] Webserver started at http://127.12.126.65:41015/ using document root <none> and password file <none>
I20250109 21:07:41.137951 28263 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:41.138001 28263 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:41.138173 28263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:07:41.139952 28263 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "b3f718ba5aab462a8a89ad217d05c03e"
format_stamp: "Formatted at 2025-01-09 21:07:41 on dist-test-slave-g0zb"
server_key: "4d4f042dc7676813ace540dccc9c62c7"
server_key_iv: "b7780acea10b144c883f2d5003e98ba6"
server_key_version: "encryptionkey@0"
I20250109 21:07:41.140384 28263 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "b3f718ba5aab462a8a89ad217d05c03e"
format_stamp: "Formatted at 2025-01-09 21:07:41 on dist-test-slave-g0zb"
server_key: "4d4f042dc7676813ace540dccc9c62c7"
server_key_iv: "b7780acea10b144c883f2d5003e98ba6"
server_key_version: "encryptionkey@0"
I20250109 21:07:41.143644 28263 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250109 21:07:41.145725 28278 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:41.146656 28263 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 21:07:41.146756 28263 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "b3f718ba5aab462a8a89ad217d05c03e"
format_stamp: "Formatted at 2025-01-09 21:07:41 on dist-test-slave-g0zb"
server_key: "4d4f042dc7676813ace540dccc9c62c7"
server_key_iv: "b7780acea10b144c883f2d5003e98ba6"
server_key_version: "encryptionkey@0"
I20250109 21:07:41.146873 28263 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:41.159319 28263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:41.161734 28263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:41.161900 28263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:41.162384 28263 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:07:41.163216 28263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:07:41.163265 28263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:41.163318 28263 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:07:41.163350 28263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:41.171376 28263 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:39075
I20250109 21:07:41.171415 28391 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:39075 every 8 connection(s)
I20250109 21:07:41.172251 28263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:07:41.173293 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 28263
I20250109 21:07:41.173417 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:07:41.173723 12793 external_mini_cluster.cc:1453] Setting key 67652e07ed4d423986cf6af6e6b648ed
Jan 09 21:07:41 dist-test-slave-g0zb krb5kdc[28156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456861, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.126.66@KRBTEST.COM; defaulting to no policy
I20250109 21:07:41.184350 28240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.173776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:37369 (local address 127.12.126.126:35275)
0109 21:07:41.173899 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:41.173903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:41.174650 (+ 747us) server_negotiation.cc:407] Connection header received
0109 21:07:41.175896 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:41.175902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:41.175964 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:41.176069 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:41.177241 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.178042 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:41.178636 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.178864 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:41.181145 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:41.181161 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:41.181164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:41.181187 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:41.182607 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:41.183052 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:41.183054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:41.183055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:41.183104 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:41.183408 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:41.183410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:41.183412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:41.183542 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:41.183628 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:41.184075 (+ 447us) server_negotiation.cc:299] Negotiation successful
0109 21:07:41.184194 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.66@KRBTEST.COM" created.
I20250109 21:07:41.184908 28394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.174119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:37369)
0109 21:07:41.174519 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:41.174559 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:41.175694 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:41.176202 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:41.176209 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:41.176561 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:41.177096 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:41.177108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.178150 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:41.178153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:41.178535 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:41.178541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.178688 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:41.179249 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:41.179271 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:41.180950 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:41.182706 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:41.182710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:41.182718 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:41.182975 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:41.183184 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:41.183186 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:41.183188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:41.183289 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:41.183641 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:41.183648 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:41.183846 (+ 198us) client_negotiation.cc:769] Sending connection context
0109 21:07:41.184002 (+ 156us) client_negotiation.cc:240] Negotiation successful
0109 21:07:41.184258 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":97,"threads_started":1}
I20250109 21:07:41.186065 28392 heartbeater.cc:346] Connected to a master server at 127.12.126.126:35275
I20250109 21:07:41.186287 28392 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:41.186913 28392 heartbeater.cc:510] Master 127.12.126.126:35275 requested a full tablet report, sending...
I20250109 21:07:41.188279 28197 ts_manager.cc:194] Registered new tserver with Master: b3f718ba5aab462a8a89ad217d05c03e (127.12.126.65:39075)
I20250109 21:07:41.189420 28197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:37369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
I20250109 21:07:41.201711 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35275
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:07:41.303681 28397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:41.303906 28397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:41.303994 28397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:41.306962 28397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:41.307019 28397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:41.307116 28397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:07:41.311064 28397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:41.312007 28397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:41.312776 28397 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:41.320645 28402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:41.320847 28405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:41.321444 28403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:41.322417 28397 server_base.cc:1034] running on GCE node
I20250109 21:07:41.322719 28397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:41.323236 28397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:41.324378 28397 hybrid_clock.cc:648] HybridClock initialized: now 1736456861324360 us; error 34 us; skew 500 ppm
Jan 09 21:07:41 dist-test-slave-g0zb krb5kdc[28156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456861, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:41.327121 28397 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:07:41.327879 28397 webserver.cc:458] Webserver started at http://127.12.126.66:41565/ using document root <none> and password file <none>
I20250109 21:07:41.328387 28397 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:41.328434 28397 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:41.328579 28397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:07:41.330278 28397 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "63a58d7214904fd9b45c2f1cf75a3680"
format_stamp: "Formatted at 2025-01-09 21:07:41 on dist-test-slave-g0zb"
server_key: "ea721f122ba6c1950019514e820fc7ca"
server_key_iv: "223efb89a09aae1d4c5ca6e020dd40c3"
server_key_version: "encryptionkey@0"
I20250109 21:07:41.330698 28397 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "63a58d7214904fd9b45c2f1cf75a3680"
format_stamp: "Formatted at 2025-01-09 21:07:41 on dist-test-slave-g0zb"
server_key: "ea721f122ba6c1950019514e820fc7ca"
server_key_iv: "223efb89a09aae1d4c5ca6e020dd40c3"
server_key_version: "encryptionkey@0"
I20250109 21:07:41.333966 28397 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20250109 21:07:41.335988 28412 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:41.336884 28397 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:07:41.336999 28397 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "63a58d7214904fd9b45c2f1cf75a3680"
format_stamp: "Formatted at 2025-01-09 21:07:41 on dist-test-slave-g0zb"
server_key: "ea721f122ba6c1950019514e820fc7ca"
server_key_iv: "223efb89a09aae1d4c5ca6e020dd40c3"
server_key_version: "encryptionkey@0"
I20250109 21:07:41.337093 28397 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:41.360682 28397 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:41.363302 28397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:41.363459 28397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:41.363934 28397 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:07:41.364671 28397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:07:41.364719 28397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:41.364773 28397 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:07:41.364806 28397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:41.372901 28397 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:44567
I20250109 21:07:41.372924 28525 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:44567 every 8 connection(s)
I20250109 21:07:41.373751 28397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:07:41.377382 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 28397
I20250109 21:07:41.377511 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:07:41.377797 12793 external_mini_cluster.cc:1453] Setting key c0583538018cebbf2a337b64a825ede0
Jan 09 21:07:41 dist-test-slave-g0zb krb5kdc[28156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456861, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:07:41.385776 28240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.375336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:53849 (local address 127.12.126.126:35275)
0109 21:07:41.375459 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:41.375462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:41.376032 (+ 570us) server_negotiation.cc:407] Connection header received
0109 21:07:41.377140 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:41.377143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:41.377189 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:41.377268 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:41.378705 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.379309 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:41.380138 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.380265 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:41.382506 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:41.382519 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:41.382522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:41.382544 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:41.384051 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:41.384495 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:41.384499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:41.384501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:41.384544 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:41.384795 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:41.384798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:41.384800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:41.384945 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:41.385030 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:41.385493 (+ 463us) server_negotiation.cc:299] Negotiation successful
0109 21:07:41.385609 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250109 21:07:41.386386 28528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.375534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:53849)
0109 21:07:41.375898 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:41.375929 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:41.376967 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:41.377452 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:41.377464 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:41.377959 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:41.378588 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:41.378604 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.379419 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:41.379423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:41.380031 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:41.380045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.380289 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:41.380879 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:41.380918 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:41.382365 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:41.384155 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:41.384160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:41.384171 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:41.384402 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:41.384633 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:41.384635 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:41.384636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:41.384712 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:41.385038 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:41.385047 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:41.385274 (+ 227us) client_negotiation.cc:769] Sending connection context
0109 21:07:41.385445 (+ 171us) client_negotiation.cc:240] Negotiation successful
0109 21:07:41.385709 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.12.126.67@KRBTEST.COM; defaulting to no policy
I20250109 21:07:41.387446 28526 heartbeater.cc:346] Connected to a master server at 127.12.126.126:35275
I20250109 21:07:41.387651 28526 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:41.388109 28526 heartbeater.cc:510] Master 127.12.126.126:35275 requested a full tablet report, sending...
I20250109 21:07:41.389055 28197 ts_manager.cc:194] Registered new tserver with Master: 63a58d7214904fd9b45c2f1cf75a3680 (127.12.126.66:44567)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.67@KRBTEST.COM" created.
I20250109 21:07:41.389569 28197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:53849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
I20250109 21:07:41.403681 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35275
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:07:41.502210 28531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:07:41.502418 28531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:07:41.502506 28531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:07:41.505360 28531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:07:41.505409 28531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:07:41.505502 28531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:07:41.509145 28531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:07:41.509972 28531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:07:41.510668 28531 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:07:41.518071 28539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:41.518258 28537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:07:41.519081 28536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:07:41.519735 28531 server_base.cc:1034] running on GCE node
I20250109 21:07:41.520072 28531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:07:41.520538 28531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:07:41.521687 28531 hybrid_clock.cc:648] HybridClock initialized: now 1736456861521667 us; error 37 us; skew 500 ppm
Jan 09 21:07:41 dist-test-slave-g0zb krb5kdc[28156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456861, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:07:41.524278 28531 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:07:41.525086 28531 webserver.cc:458] Webserver started at http://127.12.126.67:45797/ using document root <none> and password file <none>
I20250109 21:07:41.525585 28531 fs_manager.cc:362] Metadata directory not provided
I20250109 21:07:41.525642 28531 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:07:41.525799 28531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:07:41.527503 28531 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "d56ac4f18fc842ec9977c65395db3206"
format_stamp: "Formatted at 2025-01-09 21:07:41 on dist-test-slave-g0zb"
server_key: "c8481caa2dd1d37ffbe0d51e88690ae3"
server_key_iv: "5f7399b947afe57bc68d9e89c9cbf262"
server_key_version: "encryptionkey@0"
I20250109 21:07:41.527985 28531 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "d56ac4f18fc842ec9977c65395db3206"
format_stamp: "Formatted at 2025-01-09 21:07:41 on dist-test-slave-g0zb"
server_key: "c8481caa2dd1d37ffbe0d51e88690ae3"
server_key_iv: "5f7399b947afe57bc68d9e89c9cbf262"
server_key_version: "encryptionkey@0"
I20250109 21:07:41.530997 28531 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:07:41.533028 28546 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:41.533927 28531 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:07:41.534027 28531 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "d56ac4f18fc842ec9977c65395db3206"
format_stamp: "Formatted at 2025-01-09 21:07:41 on dist-test-slave-g0zb"
server_key: "c8481caa2dd1d37ffbe0d51e88690ae3"
server_key_iv: "5f7399b947afe57bc68d9e89c9cbf262"
server_key_version: "encryptionkey@0"
I20250109 21:07:41.534106 28531 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:07:41.547413 28531 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:07:41.550194 28531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:07:41.550344 28531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:07:41.550786 28531 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:07:41.551554 28531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:07:41.551602 28531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:41.551656 28531 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:07:41.551692 28531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:07:41.559926 28531 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:43475
I20250109 21:07:41.559957 28659 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:43475 every 8 connection(s)
I20250109 21:07:41.560748 28531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
Jan 09 21:07:41 dist-test-slave-g0zb krb5kdc[28156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456861, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:07:41.568460 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 28531
I20250109 21:07:41.568568 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:07:41.568862 12793 external_mini_cluster.cc:1453] Setting key e262368007fbf955d1caff34a24320c9
I20250109 21:07:41.572069 28240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.562289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:36287 (local address 127.12.126.126:35275)
0109 21:07:41.562425 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:41.562428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:41.563058 (+ 630us) server_negotiation.cc:407] Connection header received
0109 21:07:41.564234 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:41.564238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:41.564296 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:41.564379 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:41.565490 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.565956 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:41.566538 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.566718 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:41.568861 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:41.568879 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:41.568882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:41.568910 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:41.570554 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:41.571070 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:41.571073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:41.571074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:41.571118 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:41.571316 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:41.571318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:41.571319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:41.571442 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:41.571507 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:41.571871 (+ 364us) server_negotiation.cc:299] Negotiation successful
0109 21:07:41.571957 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20250109 21:07:41.572608 28662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.562515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:36287)
0109 21:07:41.562926 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:41.562969 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:41.564027 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:41.564529 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:41.564537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:41.564852 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:07:41.565350 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:41.565362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.566064 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:41.566067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:41.566425 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:41.566432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.566620 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:41.567202 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:07:41.567224 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:07:41.568706 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:07:41.570665 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:41.570671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:41.570681 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:41.570981 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:41.571190 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:07:41.571192 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:07:41.571194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:07:41.571262 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:07:41.571505 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:07:41.571509 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:07:41.571694 (+ 185us) client_negotiation.cc:769] Sending connection context
0109 21:07:41.571831 (+ 137us) client_negotiation.cc:240] Negotiation successful
0109 21:07:41.572028 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":91,"threads_started":1}
I20250109 21:07:41.573484 28660 heartbeater.cc:346] Connected to a master server at 127.12.126.126:35275
I20250109 21:07:41.573678 28660 heartbeater.cc:463] Registering TS with master...
I20250109 21:07:41.574083 28660 heartbeater.cc:510] Master 127.12.126.126:35275 requested a full tablet report, sending...
I20250109 21:07:41.574939 28197 ts_manager.cc:194] Registered new tserver with Master: d56ac4f18fc842ec9977c65395db3206 (127.12.126.67:43475)
I20250109 21:07:41.575456 28197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:36287
I20250109 21:07:41.582190 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 21:07:41.589854 28240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.583855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36194 (local address 127.12.126.126:35275)
0109 21:07:41.584001 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:41.584006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:41.584117 (+ 111us) server_negotiation.cc:407] Connection header received
0109 21:07:41.584271 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:41.584274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:41.584310 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:41.584372 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:07:41.585121 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.585554 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:41.586122 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.586311 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:41.587312 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:07:41.587325 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:07:41.587328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:07:41.587348 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:07:41.588420 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:41.588766 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:41.588770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:41.588772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:41.588811 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:07:41.588999 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:07:41.589002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:07:41.589004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:07:41.589148 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:07:41.589266 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:41.589605 (+ 339us) server_negotiation.cc:299] Negotiation successful
0109 21:07:41.589717 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250109 21:07:41.592403 28197 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:07:41.594765 28197 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:07:41.605921 28671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.601943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:39075 (local address 127.0.0.1:35834)
0109 21:07:41.602385 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:41.602418 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:41.602537 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:41.603361 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:41.603366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:41.603388 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:41.603658 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:41.603667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.604848 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:41.604850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:41.605445 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:41.605453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.605557 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:41.605611 (+ 54us) client_negotiation.cc:769] Sending connection context
0109 21:07:41.605702 (+ 91us) client_negotiation.cc:240] Negotiation successful
0109 21:07:41.605788 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":76,"threads_started":1}
I20250109 21:07:41.606112 28672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.602043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:44567 (local address 127.0.0.1:60008)
0109 21:07:41.602696 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:41.602709 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:41.602779 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:41.603622 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:41.603626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:41.603649 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:41.603878 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:41.603889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.604955 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:41.604958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:41.605799 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:41.605806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.605891 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:41.605907 (+ 16us) client_negotiation.cc:769] Sending connection context
0109 21:07:41.605950 (+ 43us) client_negotiation.cc:240] Negotiation successful
0109 21:07:41.605998 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":100,"threads_started":1}
I20250109 21:07:41.606916 28673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.602048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.12.126.65:39075)
0109 21:07:41.602880 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:41.602888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:41.602913 (+ 25us) server_negotiation.cc:407] Connection header received
0109 21:07:41.603012 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:41.603018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:41.603206 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:41.603368 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:41.603778 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.604747 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:41.605582 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.606530 (+ 948us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:41.606618 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:41.606684 (+ 66us) server_negotiation.cc:299] Negotiation successful
0109 21:07:41.606764 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":114,"threads_started":1}
I20250109 21:07:41.606938 28674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.602424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:43475 (local address 127.0.0.1:46712)
0109 21:07:41.603750 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:41.603763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:41.603846 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:41.604235 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:41.604238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:41.604256 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:41.604471 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:41.604477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.605735 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:41.605738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:41.606570 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:41.606582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.606686 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:41.606700 (+ 14us) client_negotiation.cc:769] Sending connection context
0109 21:07:41.606747 (+ 47us) client_negotiation.cc:240] Negotiation successful
0109 21:07:41.606818 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1262,"thread_start_us":88,"threads_started":1}
I20250109 21:07:41.607630 28676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.602791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46712 (local address 127.12.126.67:43475)
0109 21:07:41.603131 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:41.603136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:41.603853 (+ 717us) server_negotiation.cc:407] Connection header received
0109 21:07:41.603963 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:41.603968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:41.604099 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:41.604232 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:41.604580 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.605612 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:41.606720 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.607236 (+ 516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:41.607354 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:41.607419 (+ 65us) server_negotiation.cc:299] Negotiation successful
0109 21:07:41.607496 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":109,"threads_started":1}
I20250109 21:07:41.607767 28675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.602668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60008 (local address 127.12.126.66:44567)
0109 21:07:41.603223 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:41.603230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:41.603252 (+ 22us) server_negotiation.cc:407] Connection header received
0109 21:07:41.603332 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:41.603337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:41.603483 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:41.603625 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:41.603992 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.604848 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:41.606602 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.607362 (+ 760us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:41.607443 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:41.607497 (+ 54us) server_negotiation.cc:299] Negotiation successful
0109 21:07:41.607554 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":116,"threads_started":1}
I20250109 21:07:41.609512 28326 tablet_service.cc:1467] Processing CreateTablet for tablet bf63c0f83c7047fdb861cf19e9cfe306 (DEFAULT_TABLE table=test-table [id=6e61f01c6fe24172bfbda61edb3c29ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:07:41.609840 28594 tablet_service.cc:1467] Processing CreateTablet for tablet bf63c0f83c7047fdb861cf19e9cfe306 (DEFAULT_TABLE table=test-table [id=6e61f01c6fe24172bfbda61edb3c29ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:07:41.609839 28460 tablet_service.cc:1467] Processing CreateTablet for tablet bf63c0f83c7047fdb861cf19e9cfe306 (DEFAULT_TABLE table=test-table [id=6e61f01c6fe24172bfbda61edb3c29ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:07:41.610298 28326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf63c0f83c7047fdb861cf19e9cfe306. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:41.610656 28460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf63c0f83c7047fdb861cf19e9cfe306. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:41.610656 28594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf63c0f83c7047fdb861cf19e9cfe306. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:07:41.615428 28677 tablet_bootstrap.cc:492] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e: Bootstrap starting.
I20250109 21:07:41.616564 28678 tablet_bootstrap.cc:492] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680: Bootstrap starting.
I20250109 21:07:41.617206 28679 tablet_bootstrap.cc:492] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206: Bootstrap starting.
I20250109 21:07:41.617712 28677 tablet_bootstrap.cc:654] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e: Neither blocks nor log segments found. Creating new log.
I20250109 21:07:41.618821 28679 tablet_bootstrap.cc:654] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206: Neither blocks nor log segments found. Creating new log.
I20250109 21:07:41.618909 28678 tablet_bootstrap.cc:654] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680: Neither blocks nor log segments found. Creating new log.
I20250109 21:07:41.618924 28677 log.cc:826] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:41.619917 28679 log.cc:826] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:41.619930 28678 log.cc:826] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680: Log is configured to *not* fsync() on all Append() calls
I20250109 21:07:41.620832 28677 tablet_bootstrap.cc:492] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e: No bootstrap required, opened a new log
I20250109 21:07:41.620987 28677 ts_tablet_manager.cc:1397] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250109 21:07:41.621495 28678 tablet_bootstrap.cc:492] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680: No bootstrap required, opened a new log
I20250109 21:07:41.621495 28679 tablet_bootstrap.cc:492] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206: No bootstrap required, opened a new log
I20250109 21:07:41.621651 28678 ts_tablet_manager.cc:1397] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20250109 21:07:41.621651 28679 ts_tablet_manager.cc:1397] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250109 21:07:41.623555 28677 raft_consensus.cc:357] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } }
I20250109 21:07:41.623719 28677 raft_consensus.cc:383] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:41.623777 28677 raft_consensus.cc:738] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f718ba5aab462a8a89ad217d05c03e, State: Initialized, Role: FOLLOWER
I20250109 21:07:41.624167 28677 consensus_queue.cc:260] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } }
I20250109 21:07:41.624823 28392 heartbeater.cc:502] Master 127.12.126.126:35275 was elected leader, sending a full tablet report...
I20250109 21:07:41.624724 28678 raft_consensus.cc:357] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } }
I20250109 21:07:41.624724 28679 raft_consensus.cc:357] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } }
I20250109 21:07:41.624913 28679 raft_consensus.cc:383] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:41.624913 28678 raft_consensus.cc:383] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:07:41.624939 28677 ts_tablet_manager.cc:1428] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250109 21:07:41.624969 28678 raft_consensus.cc:738] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a58d7214904fd9b45c2f1cf75a3680, State: Initialized, Role: FOLLOWER
I20250109 21:07:41.624962 28679 raft_consensus.cc:738] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56ac4f18fc842ec9977c65395db3206, State: Initialized, Role: FOLLOWER
I20250109 21:07:41.625365 28678 consensus_queue.cc:260] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } }
I20250109 21:07:41.625363 28679 consensus_queue.cc:260] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } }
I20250109 21:07:41.626154 28678 ts_tablet_manager.cc:1428] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20250109 21:07:41.626154 28679 ts_tablet_manager.cc:1428] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250109 21:07:41.626159 28660 heartbeater.cc:502] Master 127.12.126.126:35275 was elected leader, sending a full tablet report...
I20250109 21:07:41.627044 28526 heartbeater.cc:502] Master 127.12.126.126:35275 was elected leader, sending a full tablet report...
W20250109 21:07:41.673264 28393 tablet.cc:2367] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:07:41.697700 28684 raft_consensus.cc:491] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:07:41.697885 28684 raft_consensus.cc:513] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } }
I20250109 21:07:41.698989 28684 leader_election.cc:290] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3f718ba5aab462a8a89ad217d05c03e (127.12.126.65:39075), 63a58d7214904fd9b45c2f1cf75a3680 (127.12.126.66:44567)
I20250109 21:07:41.701511 28662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.699191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:44567 (local address 127.12.126.67:44093)
0109 21:07:41.699310 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:41.699326 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:41.699410 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:41.699637 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:41.699640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:41.699661 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:41.699864 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:41.699870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.700565 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:41.700569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:41.701128 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:41.701135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.701241 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:41.701263 (+ 22us) client_negotiation.cc:769] Sending connection context
0109 21:07:41.701319 (+ 56us) client_negotiation.cc:240] Negotiation successful
0109 21:07:41.701380 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20250109 21:07:41.701974 28675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.699285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:44093 (local address 127.12.126.66:44567)
0109 21:07:41.699411 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:41.699415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:41.699427 (+ 12us) server_negotiation.cc:407] Connection header received
0109 21:07:41.699493 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:41.699495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:41.699534 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:41.699608 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:41.699986 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.700463 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:41.701248 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.701740 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:41.701784 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:41.701827 (+ 43us) server_negotiation.cc:299] Negotiation successful
0109 21:07:41.701871 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250109 21:07:41.702123 28686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.699191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:39075 (local address 127.12.126.67:40551)
0109 21:07:41.699762 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:41.699773 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0109 21:07:41.699862 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:07:41.700112 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:07:41.700115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:07:41.700130 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:07:41.700366 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:41.700372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.701026 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:07:41.701031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:07:41.701676 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:07:41.701685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.701836 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:41.701855 (+ 19us) client_negotiation.cc:769] Sending connection context
0109 21:07:41.701928 (+ 73us) client_negotiation.cc:240] Negotiation successful
0109 21:07:41.701982 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":290,"thread_start_us":93,"threads_started":1}
I20250109 21:07:41.702517 28480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf63c0f83c7047fdb861cf19e9cfe306" candidate_uuid: "d56ac4f18fc842ec9977c65395db3206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63a58d7214904fd9b45c2f1cf75a3680" is_pre_election: true
I20250109 21:07:41.702562 28673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.699285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:40551 (local address 127.12.126.65:39075)
0109 21:07:41.699414 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:41.699418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:41.699845 (+ 427us) server_negotiation.cc:407] Connection header received
0109 21:07:41.699982 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:41.699984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:41.700026 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:41.700092 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:07:41.700470 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.700917 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:41.701835 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.702344 (+ 509us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:41.702387 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:41.702437 (+ 50us) server_negotiation.cc:299] Negotiation successful
0109 21:07:41.702480 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250109 21:07:41.702780 28480 raft_consensus.cc:2463] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d56ac4f18fc842ec9977c65395db3206 in term 0.
I20250109 21:07:41.703059 28346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf63c0f83c7047fdb861cf19e9cfe306" candidate_uuid: "d56ac4f18fc842ec9977c65395db3206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3f718ba5aab462a8a89ad217d05c03e" is_pre_election: true
I20250109 21:07:41.703341 28346 raft_consensus.cc:2463] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d56ac4f18fc842ec9977c65395db3206 in term 0.
I20250109 21:07:41.703333 28549 leader_election.cc:304] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63a58d7214904fd9b45c2f1cf75a3680, d56ac4f18fc842ec9977c65395db3206; no voters:
I20250109 21:07:41.703581 28684 raft_consensus.cc:2798] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:07:41.703671 28684 raft_consensus.cc:491] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:07:41.703722 28684 raft_consensus.cc:3054] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:41.704655 28684 raft_consensus.cc:513] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } }
I20250109 21:07:41.704977 28684 leader_election.cc:290] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [CANDIDATE]: Term 1 election: Requested vote from peers b3f718ba5aab462a8a89ad217d05c03e (127.12.126.65:39075), 63a58d7214904fd9b45c2f1cf75a3680 (127.12.126.66:44567)
I20250109 21:07:41.705245 28346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf63c0f83c7047fdb861cf19e9cfe306" candidate_uuid: "d56ac4f18fc842ec9977c65395db3206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3f718ba5aab462a8a89ad217d05c03e"
I20250109 21:07:41.705317 28480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf63c0f83c7047fdb861cf19e9cfe306" candidate_uuid: "d56ac4f18fc842ec9977c65395db3206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63a58d7214904fd9b45c2f1cf75a3680"
I20250109 21:07:41.705359 28346 raft_consensus.cc:3054] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:41.705432 28480 raft_consensus.cc:3054] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:07:41.706560 28346 raft_consensus.cc:2463] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d56ac4f18fc842ec9977c65395db3206 in term 1.
I20250109 21:07:41.706560 28480 raft_consensus.cc:2463] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d56ac4f18fc842ec9977c65395db3206 in term 1.
I20250109 21:07:41.706882 28549 leader_election.cc:304] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63a58d7214904fd9b45c2f1cf75a3680, d56ac4f18fc842ec9977c65395db3206; no voters:
I20250109 21:07:41.707083 28684 raft_consensus.cc:2798] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:07:41.707275 28684 raft_consensus.cc:695] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [term 1 LEADER]: Becoming Leader. State: Replica: d56ac4f18fc842ec9977c65395db3206, State: Running, Role: LEADER
I20250109 21:07:41.707543 28684 consensus_queue.cc:237] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } }
I20250109 21:07:41.710204 28201 catalog_manager.cc:5526] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 reported cstate change: term changed from 0 to 1, leader changed from <none> to d56ac4f18fc842ec9977c65395db3206 (127.12.126.67). New cstate: current_term: 1 leader_uuid: "d56ac4f18fc842ec9977c65395db3206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } health_report { overall_health: HEALTHY } } }
I20250109 21:07:41.725610 28676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.723183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46714 (local address 127.12.126.67:43475)
0109 21:07:41.723296 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0109 21:07:41.723299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:07:41.723312 (+ 13us) server_negotiation.cc:407] Connection header received
0109 21:07:41.723385 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:07:41.723387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:07:41.723433 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:07:41.723516 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:07:41.723766 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.724225 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:07:41.724751 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:07:41.724870 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:07:41.724943 (+ 73us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:07:41.725340 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:07:41.725422 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0109 21:07:41.725483 (+ 61us) server_negotiation.cc:299] Negotiation successful
0109 21:07:41.725521 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250109 21:07:41.730321 28480 raft_consensus.cc:1270] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680 [term 1 FOLLOWER]: Refusing update from remote peer d56ac4f18fc842ec9977c65395db3206: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:07:41.730321 28346 raft_consensus.cc:1270] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Refusing update from remote peer d56ac4f18fc842ec9977c65395db3206: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:07:41.730937 28684 consensus_queue.cc:1035] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:07:41.731122 28687 consensus_queue.cc:1035] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:07:41.736909 28690 mvcc.cc:204] Tried to move back new op lower bound from 7112527305641480192 to 7112527305556336640. Current Snapshot: MvccSnapshot[applied={T|T < 7112527305641480192}]
I20250109 21:07:41.738979 28691 mvcc.cc:204] Tried to move back new op lower bound from 7112527305641480192 to 7112527305556336640. Current Snapshot: MvccSnapshot[applied={T|T < 7112527305641480192}]
I20250109 21:07:41.744695 28201 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36194:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:07:41.744827 28201 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36194:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:07:41.746814 28201 catalog_manager.cc:5813] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: Sending DeleteTablet for 3 replicas of tablet bf63c0f83c7047fdb861cf19e9cfe306
I20250109 21:07:41.747454 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet bf63c0f83c7047fdb861cf19e9cfe306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:07:41 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:60008
I20250109 21:07:41.747454 28326 tablet_service.cc:1514] Processing DeleteTablet for tablet bf63c0f83c7047fdb861cf19e9cfe306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:07:41 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:35834
I20250109 21:07:41.747629 28594 tablet_service.cc:1514] Processing DeleteTablet for tablet bf63c0f83c7047fdb861cf19e9cfe306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:07:41 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:46712
I20250109 21:07:41.748001 28701 tablet_replica.cc:331] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e: stopping tablet replica
I20250109 21:07:41.748068 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 28263
I20250109 21:07:41.748075 28700 tablet_replica.cc:331] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680: stopping tablet replica
I20250109 21:07:41.748266 28701 raft_consensus.cc:2238] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:07:41.749061 28700 raft_consensus.cc:2238] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:07:41.749336 28700 raft_consensus.cc:2267] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:07:41.751389 28702 tablet_replica.cc:331] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206: stopping tablet replica
I20250109 21:07:41.751575 28702 raft_consensus.cc:2238] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:07:41.751863 28702 raft_consensus.cc:2267] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:07:41.752873 28702 ts_tablet_manager.cc:1905] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:07:41.754778 28702 ts_tablet_manager.cc:1918] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:07:41.754880 28702 log.cc:1198] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/bf63c0f83c7047fdb861cf19e9cfe306
I20250109 21:07:41.755890 28700 ts_tablet_manager.cc:1905] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 21:07:41.755939 28185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.126.65:39075 (error 108)
I20250109 21:07:41.756181 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 28397
I20250109 21:07:41.756839 28702 ts_tablet_manager.cc:1939] T bf63c0f83c7047fdb861cf19e9cfe306 P d56ac4f18fc842ec9977c65395db3206: Deleting consensus metadata
I20250109 21:07:41.757035 28674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.756722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:39075 (local address 127.0.0.1:35838)
0109 21:07:41.756891 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:41.756960 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.65:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20250109 21:07:41.757252 28185 catalog_manager.cc:4615] TS b3f718ba5aab462a8a89ad217d05c03e (127.12.126.65:39075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf63c0f83c7047fdb861cf19e9cfe306: Network error: Client connection negotiation failed: client connection to 127.12.126.65:39075: connect: Connection refused (error 111)
I20250109 21:07:41.757871 28186 catalog_manager.cc:4872] TS d56ac4f18fc842ec9977c65395db3206 (127.12.126.67:43475): tablet bf63c0f83c7047fdb861cf19e9cfe306 (table test-table [id=6e61f01c6fe24172bfbda61edb3c29ca]) successfully deleted
I20250109 21:07:41.763950 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 28531
I20250109 21:07:41.765656 28674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:07:41.765442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:44567 (local address 127.0.0.1:60016)
0109 21:07:41.765556 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0109 21:07:41.765595 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.66:44567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20250109 21:07:41.768464 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 28169
I20250109 21:08:01.774955 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:35275
--webserver_interface=127.12.126.126
--webserver_port=46205
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:35275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:01.880455 28704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:01.880700 28704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:01.880774 28704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:01.884081 28704 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20250109 21:08:01.884147 28704 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20250109 21:08:01.884167 28704 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20250109 21:08:01.884182 28704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:08:01.884203 28704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:01.884225 28704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:01.884266 28704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:08:01.884290 28704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:08:01.888762 28704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:35275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:35275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=46205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:01.889906 28704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:01.890761 28704 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250109 21:08:01.903069 28704 server_base.cc:1034] running on GCE node
W20250109 21:08:01.903558 28713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:01.903790 28711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:01.904379 28710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:01.904933 28704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:01.905896 28704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:01.907049 28704 hybrid_clock.cc:648] HybridClock initialized: now 1736456881907031 us; error 33 us; skew 500 ppm
Jan 09 21:08:01 dist-test-slave-g0zb krb5kdc[28156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456881, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:01.909857 28704 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:08:01.910763 28704 webserver.cc:458] Webserver started at http://127.12.126.126:46205/ using document root <none> and password file <none>
I20250109 21:08:01.911370 28704 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:01.911438 28704 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:01.914971 28704 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20250109 21:08:01.916854 28720 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:01.917927 28704 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:01.918035 28704 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "cf5434de2d4448068da2c603fb583b04"
format_stamp: "Formatted at 2025-01-09 21:07:40 on dist-test-slave-g0zb"
server_key: "33320c6b3e8aa2962f9f4bfff1b9b7b3"
server_key_iv: "74abe7bf2c0fffc09886a847e60e63a3"
server_key_version: "encryptionkey@0"
I20250109 21:08:01.918515 28704 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:01.936620 28704 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:01.939970 28704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:01.940145 28704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:01.947613 28704 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:35275
I20250109 21:08:01.947914 28772 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:35275 every 8 connection(s)
I20250109 21:08:01.948810 28704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:08:01.950757 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 28704
I20250109 21:08:01.951231 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:39075
--local_ip_for_outbound_sockets=127.12.126.65
--tserver_master_addrs=127.12.126.126:35275
--webserver_port=41015
--webserver_interface=127.12.126.65
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:08:01.953166 28773 sys_catalog.cc:263] Verifying existing consensus state
I20250109 21:08:01.954871 28773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: Bootstrap starting.
I20250109 21:08:01.977964 28773 log.cc:826] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:01.988623 28773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:08:01.989070 28773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: Bootstrap complete.
I20250109 21:08:01.993810 28773 raft_consensus.cc:357] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5434de2d4448068da2c603fb583b04" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35275 } }
I20250109 21:08:01.994249 28773 raft_consensus.cc:738] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5434de2d4448068da2c603fb583b04, State: Initialized, Role: FOLLOWER
I20250109 21:08:01.994848 28773 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5434de2d4448068da2c603fb583b04" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35275 } }
I20250109 21:08:01.994992 28773 raft_consensus.cc:397] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:08:01.995054 28773 raft_consensus.cc:491] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:08:01.995157 28773 raft_consensus.cc:3054] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 1 FOLLOWER]: Advancing to term 2
I20250109 21:08:01.997272 28773 raft_consensus.cc:513] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5434de2d4448068da2c603fb583b04" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35275 } }
I20250109 21:08:01.997648 28773 leader_election.cc:304] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf5434de2d4448068da2c603fb583b04; no voters:
I20250109 21:08:01.998028 28773 leader_election.cc:290] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250109 21:08:01.998167 28777 raft_consensus.cc:2798] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 2 FOLLOWER]: Leader election won for term 2
I20250109 21:08:01.998414 28777 raft_consensus.cc:695] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 2 LEADER]: Becoming Leader. State: Replica: cf5434de2d4448068da2c603fb583b04, State: Running, Role: LEADER
I20250109 21:08:01.998757 28777 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5434de2d4448068da2c603fb583b04" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35275 } }
I20250109 21:08:01.999241 28773 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:08:02.000242 28778 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cf5434de2d4448068da2c603fb583b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5434de2d4448068da2c603fb583b04" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35275 } } }
I20250109 21:08:02.000384 28778 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:02.000242 28779 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf5434de2d4448068da2c603fb583b04. Latest consensus state: current_term: 2 leader_uuid: "cf5434de2d4448068da2c603fb583b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5434de2d4448068da2c603fb583b04" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35275 } } }
I20250109 21:08:02.000470 28779 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:02.000754 28786 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:08:02.005640 28786 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:08:02.006476 28786 catalog_manager.cc:1260] Loaded cluster ID: 2f406a50bda84df8bbf80dc3ab5f3401
I20250109 21:08:02.006543 28786 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:08:02.009193 28786 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:08:02.009961 28786 catalog_manager.cc:5910] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: Loaded TSK: 0
I20250109 21:08:02.033990 28786 catalog_manager.cc:5899] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: Generated new TSK 1
I20250109 21:08:02.034300 28786 catalog_manager.cc:1515] Initializing in-progress tserver states...
W20250109 21:08:02.081174 28775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:02.081454 28775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:02.081543 28775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:02.084874 28775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:02.084950 28775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:02.085063 28775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:08:02.089442 28775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:39075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=41015
--tserver_master_addrs=127.12.126.126:35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:02.090552 28775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:02.091454 28775 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:02.100716 28800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:02.100747 28801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:02.101344 28803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:02.102406 28775 server_base.cc:1034] running on GCE node
I20250109 21:08:02.102929 28775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:02.103647 28775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:02.104826 28775 hybrid_clock.cc:648] HybridClock initialized: now 1736456882104809 us; error 32 us; skew 500 ppm
Jan 09 21:08:02 dist-test-slave-g0zb krb5kdc[28156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456882, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:02.108157 28775 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:08:02.109187 28775 webserver.cc:458] Webserver started at http://127.12.126.65:41015/ using document root <none> and password file <none>
I20250109 21:08:02.109756 28775 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:02.109831 28775 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:02.113574 28775 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20250109 21:08:02.115651 28810 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:02.116732 28775 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:02.116843 28775 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "b3f718ba5aab462a8a89ad217d05c03e"
format_stamp: "Formatted at 2025-01-09 21:07:41 on dist-test-slave-g0zb"
server_key: "4d4f042dc7676813ace540dccc9c62c7"
server_key_iv: "b7780acea10b144c883f2d5003e98ba6"
server_key_version: "encryptionkey@0"
I20250109 21:08:02.117288 28775 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:02.140915 28775 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:02.144042 28775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:02.144224 28775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:02.144742 28775 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:02.145781 28817 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250109 21:08:02.149423 28775 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250109 21:08:02.149479 28775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20250109 21:08:02.149525 28775 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250109 21:08:02.151011 28775 ts_tablet_manager.cc:610] Registered 1 tablets
I20250109 21:08:02.151068 28775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20250109 21:08:02.151315 28817 tablet_bootstrap.cc:492] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e: Bootstrap starting.
I20250109 21:08:02.161607 28775 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:39075
I20250109 21:08:02.161856 28924 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:39075 every 8 connection(s)
I20250109 21:08:02.163168 28775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:08:02.170202 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 28775
I20250109 21:08:02.170739 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:44567
--local_ip_for_outbound_sockets=127.12.126.66
--tserver_master_addrs=127.12.126.126:35275
--webserver_port=41565
--webserver_interface=127.12.126.66
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:08:02.171926 28817 log.cc:826] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e: Log is configured to *not* fsync() on all Append() calls
Jan 09 21:08:02 dist-test-slave-g0zb krb5kdc[28156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456882, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:08:02.181087 28927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.165315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:35199 (local address 127.12.126.126:35275)
0109 21:08:02.165743 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:02.165756 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:02.166285 (+ 529us) server_negotiation.cc:407] Connection header received
0109 21:08:02.167633 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:02.167648 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:02.168057 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:02.168506 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:02.170077 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.171363 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.172297 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.172558 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.175602 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:02.175627 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:02.175639 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:02.175680 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:02.177897 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:02.178521 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:02.178527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:02.178532 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:02.178625 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:02.178994 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:02.178997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:02.179000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:02.179363 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:02.179465 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:02.180147 (+ 682us) server_negotiation.cc:299] Negotiation successful
0109 21:08:02.180423 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":117,"threads_started":1}
I20250109 21:08:02.181460 28817 tablet_bootstrap.cc:492] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:08:02.182086 28817 tablet_bootstrap.cc:492] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e: Bootstrap complete.
I20250109 21:08:02.182706 28928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.165376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:35199)
0109 21:08:02.166085 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:02.166153 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:02.167162 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:02.168592 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:02.168605 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:02.169165 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:02.169883 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.169922 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.171522 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.171525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:02.172171 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.172179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.172793 (+ 614us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.173475 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:08:02.173505 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:02.175402 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:02.178037 (+ 2635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:02.178043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:02.178052 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:02.178392 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:02.178740 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:02.178743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:02.178745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:02.178888 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:02.179546 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:02.179551 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:02.179804 (+ 253us) client_negotiation.cc:769] Sending connection context
0109 21:08:02.181739 (+ 1935us) client_negotiation.cc:240] Negotiation successful
0109 21:08:02.182040 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":144,"threads_started":1}
I20250109 21:08:02.183609 28817 ts_tablet_manager.cc:1397] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.011s
I20250109 21:08:02.184857 28925 heartbeater.cc:346] Connected to a master server at 127.12.126.126:35275
I20250109 21:08:02.185074 28925 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:02.185597 28925 heartbeater.cc:510] Master 127.12.126.126:35275 requested a full tablet report, sending...
I20250109 21:08:02.186549 28817 raft_consensus.cc:357] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } }
I20250109 21:08:02.187145 28817 raft_consensus.cc:738] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f718ba5aab462a8a89ad217d05c03e, State: Initialized, Role: FOLLOWER
I20250109 21:08:02.187688 28817 consensus_queue.cc:260] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } }
I20250109 21:08:02.187780 28736 ts_manager.cc:194] Registered new tserver with Master: b3f718ba5aab462a8a89ad217d05c03e (127.12.126.65:39075)
I20250109 21:08:02.188699 28817 ts_tablet_manager.cc:1428] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:08:02.191088 28736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:35199
I20250109 21:08:02.193396 28925 heartbeater.cc:502] Master 127.12.126.126:35275 was elected leader, sending a full tablet report...
Jan 09 21:08:02 dist-test-slave-g0zb krb5kdc[28156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456881, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for kudu/127.12.126.65@KRBTEST.COM
I20250109 21:08:02.200966 28932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.190404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:39075 (local address 127.0.0.1:50858)
0109 21:08:02.191144 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:02.191181 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:02.191395 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:02.192081 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:02.192086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:02.192572 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:02.192904 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.192919 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.194897 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.194900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:02.195398 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.195407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.195558 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.196158 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:08:02.196175 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:02.197620 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:02.199487 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:02.199494 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:02.199499 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:02.199703 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:02.199948 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:02.199953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:02.199955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:02.200001 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:02.200373 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:02.200379 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:02.200481 (+ 102us) client_negotiation.cc:769] Sending connection context
0109 21:08:02.200606 (+ 125us) client_negotiation.cc:240] Negotiation successful
0109 21:08:02.200771 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":113,"threads_started":1}
I20250109 21:08:02.201088 28933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.190514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.12.126.65:39075)
0109 21:08:02.190890 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:02.190896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:02.191286 (+ 390us) server_negotiation.cc:407] Connection header received
0109 21:08:02.191576 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:02.191581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:02.191752 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:02.191849 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:02.193052 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.193926 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.195549 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.195705 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.197740 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:02.197759 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:02.197768 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:02.197791 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:02.199377 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:02.199810 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:02.199815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:02.199816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:02.199864 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:02.200080 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:02.200084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:02.200086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:02.200268 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:02.200390 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:02.200719 (+ 329us) server_negotiation.cc:299] Negotiation successful
0109 21:08:02.200860 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":100,"threads_started":1}
I20250109 21:08:02.202210 28859 tablet_service.cc:1514] Processing DeleteTablet for tablet bf63c0f83c7047fdb861cf19e9cfe306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:07:41 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:50858
I20250109 21:08:02.203023 28934 tablet_replica.cc:331] stopping tablet replica
I20250109 21:08:02.203321 28934 raft_consensus.cc:2238] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:08:02.203575 28934 raft_consensus.cc:2267] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:02.204599 28934 ts_tablet_manager.cc:1905] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:08:02.208480 28934 ts_tablet_manager.cc:1918] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:08:02.208578 28934 log.cc:1198] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/wal/wals/bf63c0f83c7047fdb861cf19e9cfe306
I20250109 21:08:02.208894 28934 ts_tablet_manager.cc:1939] T bf63c0f83c7047fdb861cf19e9cfe306 P b3f718ba5aab462a8a89ad217d05c03e: Deleting consensus metadata
I20250109 21:08:02.210654 28721 catalog_manager.cc:4872] TS b3f718ba5aab462a8a89ad217d05c03e (127.12.126.65:39075): tablet bf63c0f83c7047fdb861cf19e9cfe306 (table test-table [id=6e61f01c6fe24172bfbda61edb3c29ca]) successfully deleted
W20250109 21:08:02.287755 28929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:02.288004 28929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:02.288061 28929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:02.291177 28929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:02.291237 28929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:02.291312 28929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:08:02.295418 28929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:44567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=41565
--tserver_master_addrs=127.12.126.126:35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:02.296401 28929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:02.297343 28929 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:02.305965 28939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:02.306682 28942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:02.306914 28940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:02.307667 28929 server_base.cc:1034] running on GCE node
I20250109 21:08:02.308058 28929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:02.308593 28929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:02.309749 28929 hybrid_clock.cc:648] HybridClock initialized: now 1736456882309716 us; error 48 us; skew 500 ppm
Jan 09 21:08:02 dist-test-slave-g0zb krb5kdc[28156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456882, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:02.312443 28929 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:08:02.313284 28929 webserver.cc:458] Webserver started at http://127.12.126.66:41565/ using document root <none> and password file <none>
I20250109 21:08:02.313797 28929 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:02.313866 28929 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:02.317399 28929 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:02.319419 28949 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:02.320430 28929 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:02.320525 28929 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "63a58d7214904fd9b45c2f1cf75a3680"
format_stamp: "Formatted at 2025-01-09 21:07:41 on dist-test-slave-g0zb"
server_key: "ea721f122ba6c1950019514e820fc7ca"
server_key_iv: "223efb89a09aae1d4c5ca6e020dd40c3"
server_key_version: "encryptionkey@0"
I20250109 21:08:02.320920 28929 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:02.332509 28929 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:02.335402 28929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:02.335541 28929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:02.335978 28929 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:02.336884 28956 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250109 21:08:02.340526 28929 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250109 21:08:02.340596 28929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20250109 21:08:02.340683 28929 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250109 21:08:02.342204 28929 ts_tablet_manager.cc:610] Registered 1 tablets
I20250109 21:08:02.342249 28929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20250109 21:08:02.342533 28956 tablet_bootstrap.cc:492] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680: Bootstrap starting.
I20250109 21:08:02.348018 28956 log.cc:826] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:02.351835 28929 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:44567
I20250109 21:08:02.352046 29064 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:44567 every 8 connection(s)
I20250109 21:08:02.352761 28929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:08:02.355417 28956 tablet_bootstrap.cc:492] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:08:02.355839 28956 tablet_bootstrap.cc:492] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680: Bootstrap complete.
I20250109 21:08:02.356161 28956 ts_tablet_manager.cc:1397] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20250109 21:08:02.358158 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 28929
I20250109 21:08:02.358647 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:43475
--local_ip_for_outbound_sockets=127.12.126.67
--tserver_master_addrs=127.12.126.126:35275
--webserver_port=45797
--webserver_interface=127.12.126.67
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:08:02.358600 28956 raft_consensus.cc:357] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } }
I20250109 21:08:02.359036 28956 raft_consensus.cc:738] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a58d7214904fd9b45c2f1cf75a3680, State: Initialized, Role: FOLLOWER
I20250109 21:08:02.359475 28956 consensus_queue.cc:260] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } }
I20250109 21:08:02.360327 28956 ts_tablet_manager.cc:1428] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
Jan 09 21:08:02 dist-test-slave-g0zb krb5kdc[28156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456882, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:08:02.367221 28927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.354505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:49511 (local address 127.12.126.126:35275)
0109 21:08:02.354658 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:02.354664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:02.355214 (+ 550us) server_negotiation.cc:407] Connection header received
0109 21:08:02.356217 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:02.356223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:02.356287 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:02.356368 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:02.357944 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.358740 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.360353 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.360541 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.363285 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:02.363314 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:02.363318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:02.363359 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:02.365184 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:02.365652 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:02.365658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:02.365662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:02.365722 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:02.365972 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:02.365975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:02.365978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:02.366215 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:02.366336 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:02.366788 (+ 452us) server_negotiation.cc:299] Negotiation successful
0109 21:08:02.366967 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250109 21:08:02.367676 29067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.354537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:49511)
0109 21:08:02.355049 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:02.355104 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:02.355986 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:02.356536 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:02.356546 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:02.357057 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:02.357779 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.357798 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.358931 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.358937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:02.360186 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.360201 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.360700 (+ 499us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.361311 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:08:02.361336 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:02.363085 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:02.365306 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:02.365310 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:02.365320 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:02.365553 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:02.365809 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:02.365811 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:02.365812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:02.365884 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:02.366320 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:02.366325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:02.366549 (+ 224us) client_negotiation.cc:769] Sending connection context
0109 21:08:02.366712 (+ 163us) client_negotiation.cc:240] Negotiation successful
0109 21:08:02.366973 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":135,"threads_started":1}
I20250109 21:08:02.368805 29065 heartbeater.cc:346] Connected to a master server at 127.12.126.126:35275
I20250109 21:08:02.369047 29065 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:02.369660 29065 heartbeater.cc:510] Master 127.12.126.126:35275 requested a full tablet report, sending...
I20250109 21:08:02.371587 28736 ts_manager.cc:194] Registered new tserver with Master: 63a58d7214904fd9b45c2f1cf75a3680 (127.12.126.66:44567)
I20250109 21:08:02.372778 28736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:49511
I20250109 21:08:02.374833 29065 heartbeater.cc:502] Master 127.12.126.126:35275 was elected leader, sending a full tablet report...
Jan 09 21:08:02 dist-test-slave-g0zb krb5kdc[28156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456881, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for kudu/127.12.126.66@KRBTEST.COM
I20250109 21:08:02.381352 28932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.372449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:44567 (local address 127.0.0.1:42494)
0109 21:08:02.372617 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:02.372634 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:02.372713 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:02.373469 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:02.373472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:02.373661 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:02.373885 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.373894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.375201 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.375207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:02.375636 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.375646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.375771 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.376289 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:08:02.376305 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:02.377731 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:02.380060 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:02.380063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:02.380065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:02.380251 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:02.380500 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:02.380502 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:02.380504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:02.380550 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:02.380950 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:02.380952 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:02.381029 (+ 77us) client_negotiation.cc:769] Sending connection context
0109 21:08:02.381122 (+ 93us) client_negotiation.cc:240] Negotiation successful
0109 21:08:02.381236 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20250109 21:08:02.381567 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.372690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42494 (local address 127.12.126.66:44567)
0109 21:08:02.373011 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:02.373017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:02.373041 (+ 24us) server_negotiation.cc:407] Connection header received
0109 21:08:02.373132 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:02.373139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:02.373328 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:02.373461 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:02.374021 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.375091 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.375761 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.375978 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.377864 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:02.377891 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:02.377902 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:02.377928 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:02.379931 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:02.380354 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:02.380359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:02.380361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:02.380417 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:02.380623 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:02.380626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:02.380628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:02.380832 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:02.380958 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:02.381233 (+ 275us) server_negotiation.cc:299] Negotiation successful
0109 21:08:02.381400 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":110,"threads_started":1}
I20250109 21:08:02.382476 28998 tablet_service.cc:1514] Processing DeleteTablet for tablet bf63c0f83c7047fdb861cf19e9cfe306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:07:41 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:42494
I20250109 21:08:02.383431 29071 tablet_replica.cc:331] stopping tablet replica
I20250109 21:08:02.383780 29071 raft_consensus.cc:2238] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:08:02.384029 29071 raft_consensus.cc:2267] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:02.384907 29071 ts_tablet_manager.cc:1905] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:08:02.388136 29071 ts_tablet_manager.cc:1918] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:08:02.388226 29071 log.cc:1198] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/bf63c0f83c7047fdb861cf19e9cfe306
I20250109 21:08:02.388494 29071 ts_tablet_manager.cc:1939] T bf63c0f83c7047fdb861cf19e9cfe306 P 63a58d7214904fd9b45c2f1cf75a3680: Deleting consensus metadata
I20250109 21:08:02.389885 28723 catalog_manager.cc:4872] TS 63a58d7214904fd9b45c2f1cf75a3680 (127.12.126.66:44567): tablet bf63c0f83c7047fdb861cf19e9cfe306 (table test-table [id=6e61f01c6fe24172bfbda61edb3c29ca]) successfully deleted
W20250109 21:08:02.470042 29069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:02.470269 29069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:02.470327 29069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:02.473417 29069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:02.473475 29069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:02.473565 29069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:08:02.477710 29069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:43475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=45797
--tserver_master_addrs=127.12.126.126:35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:02.478773 29069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:02.479629 29069 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:02.488080 29076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:02.488821 29079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:02.489068 29077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:02.489768 29069 server_base.cc:1034] running on GCE node
I20250109 21:08:02.490139 29069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:02.490645 29069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:02.491808 29069 hybrid_clock.cc:648] HybridClock initialized: now 1736456882491793 us; error 55 us; skew 500 ppm
Jan 09 21:08:02 dist-test-slave-g0zb krb5kdc[28156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456882, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:02.494490 29069 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:08:02.495409 29069 webserver.cc:458] Webserver started at http://127.12.126.67:45797/ using document root <none> and password file <none>
I20250109 21:08:02.495950 29069 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:02.496026 29069 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:02.499447 29069 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:08:02.501456 29086 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:02.502439 29069 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 21:08:02.502543 29069 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "d56ac4f18fc842ec9977c65395db3206"
format_stamp: "Formatted at 2025-01-09 21:07:41 on dist-test-slave-g0zb"
server_key: "c8481caa2dd1d37ffbe0d51e88690ae3"
server_key_iv: "5f7399b947afe57bc68d9e89c9cbf262"
server_key_version: "encryptionkey@0"
I20250109 21:08:02.503005 29069 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:02.513881 29069 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:02.516808 29069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:02.516975 29069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:02.517485 29069 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:02.518249 29069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:02.518297 29069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:02.518358 29069 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:02.518385 29069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:02.526695 29069 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:43475
I20250109 21:08:02.526736 29199 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:43475 every 8 connection(s)
I20250109 21:08:02.527722 29069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:08:02.534991 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 29069
Jan 09 21:08:02 dist-test-slave-g0zb krb5kdc[28156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456882, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:08:02.540201 28927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.529402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:37369 (local address 127.12.126.126:35275)
0109 21:08:02.529530 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:02.529534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:02.530172 (+ 638us) server_negotiation.cc:407] Connection header received
0109 21:08:02.531451 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:02.531454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:02.531522 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:02.531601 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:02.532917 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.533490 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.534124 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.534302 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.536868 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:02.536889 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:02.536892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:02.536919 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:02.538298 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:02.538795 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:02.538832 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:02.538835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:02.538881 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:02.539147 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:02.539150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:02.539152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:02.539339 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:02.539456 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:02.539905 (+ 449us) server_negotiation.cc:299] Negotiation successful
0109 21:08:02.540023 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45,"spinlock_wait_cycles":768}
I20250109 21:08:02.540798 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.529621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:37369)
0109 21:08:02.530037 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:02.530071 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:02.531240 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:02.531767 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:02.531775 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:02.532160 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:02.532775 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.532791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.533599 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.533603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:02.534015 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.534023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.534229 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.534893 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:08:02.534917 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:02.536733 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:02.538398 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:02.538403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:02.538414 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:02.538683 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:02.538966 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:02.538969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:02.538970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:02.539050 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:02.539436 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:02.539440 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:02.539649 (+ 209us) client_negotiation.cc:769] Sending connection context
0109 21:08:02.539822 (+ 173us) client_negotiation.cc:240] Negotiation successful
0109 21:08:02.540105 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":99,"threads_started":1}
I20250109 21:08:02.541971 29200 heartbeater.cc:346] Connected to a master server at 127.12.126.126:35275
I20250109 21:08:02.542207 29200 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:02.542724 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.535649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48568 (local address 127.12.126.126:35275)
0109 21:08:02.535866 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:02.535869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:02.535968 (+ 99us) server_negotiation.cc:407] Connection header received
0109 21:08:02.536118 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:02.536120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:02.536156 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:02.536233 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:02.537076 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.537774 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.538435 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.538651 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.539791 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:02.539811 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:02.539813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:02.539834 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:02.541358 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:02.541717 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:02.541723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:02.541724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:02.541764 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:02.541996 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:02.542001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:02.542003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:02.542168 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:02.542314 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:02.542506 (+ 192us) server_negotiation.cc:299] Negotiation successful
0109 21:08:02.542598 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":57,"threads_started":1}
I20250109 21:08:02.542843 29200 heartbeater.cc:510] Master 127.12.126.126:35275 requested a full tablet report, sending...
I20250109 21:08:02.543849 28736 ts_manager.cc:194] Registered new tserver with Master: d56ac4f18fc842ec9977c65395db3206 (127.12.126.67:43475)
I20250109 21:08:02.544476 28736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:37369
I20250109 21:08:02.544623 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 21:08:02.547981 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.545197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48576 (local address 127.12.126.126:35275)
0109 21:08:02.545314 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:02.545319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:02.545451 (+ 132us) server_negotiation.cc:407] Connection header received
0109 21:08:02.545578 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:02.545581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:02.545677 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:02.545786 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:08:02.546101 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.546775 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.547386 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.547517 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.547587 (+ 70us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:08:02.547708 (+ 121us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0109 21:08:02.547893 (+ 185us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48576: token expired
Metrics: {"server-negotiator.queue_time_us":43}
W20250109 21:08:02.548036 29204 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48576: token expired
W20250109 21:08:02.548358 28664 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.126.126:35275)
I20250109 21:08:02.554162 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.548654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.12.126.126:35275)
0109 21:08:02.548755 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:02.548758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:02.548769 (+ 11us) server_negotiation.cc:407] Connection header received
0109 21:08:02.548836 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:02.548838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:02.548871 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:02.548933 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:02.549511 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.549932 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.550454 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.550559 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.551568 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:02.551580 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:02.551582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:02.551601 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:02.552976 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:02.553341 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:02.553343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:02.553345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:02.553382 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:02.553535 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:02.553537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:02.553538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:02.553654 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:02.553744 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:02.553897 (+ 153us) server_negotiation.cc:299] Negotiation successful
0109 21:08:02.553983 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250109 21:08:02.556372 28736 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:08:02.559152 28736 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:08:02.566533 28998 tablet_service.cc:1467] Processing CreateTablet for tablet d8ebafb426bc42a7932289548e95c04b (DEFAULT_TABLE table=test-table [id=af14e996f29647d79641360d4f97bc8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:02.566533 28859 tablet_service.cc:1467] Processing CreateTablet for tablet d8ebafb426bc42a7932289548e95c04b (DEFAULT_TABLE table=test-table [id=af14e996f29647d79641360d4f97bc8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:02.567137 28859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ebafb426bc42a7932289548e95c04b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:02.567137 28998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ebafb426bc42a7932289548e95c04b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:02.568900 28932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.565502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:43475 (local address 127.0.0.1:49130)
0109 21:08:02.565759 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:02.565782 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:02.565854 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:02.566517 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:02.566521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:02.566539 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:02.566768 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.566776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.567953 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.567958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:02.568607 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.568617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.568689 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.568726 (+ 37us) client_negotiation.cc:769] Sending connection context
0109 21:08:02.568757 (+ 31us) client_negotiation.cc:240] Negotiation successful
0109 21:08:02.568806 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20250109 21:08:02.569795 28817 tablet_bootstrap.cc:492] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e: Bootstrap starting.
I20250109 21:08:02.569970 29209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.565700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.12.126.67:43475)
0109 21:08:02.566087 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:02.566094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:02.566119 (+ 25us) server_negotiation.cc:407] Connection header received
0109 21:08:02.566210 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:02.566214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:02.566364 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:02.566514 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:02.566920 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.567824 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.569218 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.569570 (+ 352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.569670 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:02.569746 (+ 76us) server_negotiation.cc:299] Negotiation successful
0109 21:08:02.569839 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":151,"threads_started":1}
I20250109 21:08:02.570684 28817 tablet_bootstrap.cc:654] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:02.570729 28956 tablet_bootstrap.cc:492] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680: Bootstrap starting.
I20250109 21:08:02.571684 28956 tablet_bootstrap.cc:654] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:02.571751 28817 tablet_bootstrap.cc:492] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e: No bootstrap required, opened a new log
I20250109 21:08:02.571812 28817 ts_tablet_manager.cc:1397] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250109 21:08:02.572361 28817 raft_consensus.cc:357] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } }
I20250109 21:08:02.572444 28817 raft_consensus.cc:383] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:02.572471 28817 raft_consensus.cc:738] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f718ba5aab462a8a89ad217d05c03e, State: Initialized, Role: FOLLOWER
I20250109 21:08:02.572595 28817 consensus_queue.cc:260] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } }
I20250109 21:08:02.572896 28817 ts_tablet_manager.cc:1428] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 21:08:02.572743 29134 tablet_service.cc:1467] Processing CreateTablet for tablet d8ebafb426bc42a7932289548e95c04b (DEFAULT_TABLE table=test-table [id=af14e996f29647d79641360d4f97bc8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:02.573304 28956 tablet_bootstrap.cc:492] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680: No bootstrap required, opened a new log
I20250109 21:08:02.573379 28956 ts_tablet_manager.cc:1397] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250109 21:08:02.573746 29134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ebafb426bc42a7932289548e95c04b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:02.573817 28956 raft_consensus.cc:357] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } }
I20250109 21:08:02.573910 28956 raft_consensus.cc:383] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:02.573944 28956 raft_consensus.cc:738] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a58d7214904fd9b45c2f1cf75a3680, State: Initialized, Role: FOLLOWER
I20250109 21:08:02.574069 28956 consensus_queue.cc:260] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } }
I20250109 21:08:02.574313 28956 ts_tablet_manager.cc:1428] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 21:08:02.580235 29212 tablet_bootstrap.cc:492] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206: Bootstrap starting.
I20250109 21:08:02.582257 29212 tablet_bootstrap.cc:654] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:02.583359 29212 log.cc:826] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:02.584936 29212 tablet_bootstrap.cc:492] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206: No bootstrap required, opened a new log
I20250109 21:08:02.585114 29212 ts_tablet_manager.cc:1397] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:08:02.587723 29212 raft_consensus.cc:357] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } }
I20250109 21:08:02.587904 29212 raft_consensus.cc:383] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:02.587941 29212 raft_consensus.cc:738] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56ac4f18fc842ec9977c65395db3206, State: Initialized, Role: FOLLOWER
I20250109 21:08:02.588282 29212 consensus_queue.cc:260] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } }
I20250109 21:08:02.588907 29200 heartbeater.cc:502] Master 127.12.126.126:35275 was elected leader, sending a full tablet report...
I20250109 21:08:02.589043 29212 ts_tablet_manager.cc:1428] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250109 21:08:02.603718 29066 tablet.cc:2367] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250109 21:08:02.664530 28926 tablet.cc:2367] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250109 21:08:02.778632 29201 tablet.cc:2367] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:08:02.929127 28931 raft_consensus.cc:491] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:08:02.929358 28931 raft_consensus.cc:513] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } }
I20250109 21:08:02.930835 28931 leader_election.cc:290] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d56ac4f18fc842ec9977c65395db3206 (127.12.126.67:43475), 63a58d7214904fd9b45c2f1cf75a3680 (127.12.126.66:44567)
I20250109 21:08:02.934108 29068 raft_consensus.cc:491] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:08:02.934274 29068 raft_consensus.cc:513] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } }
I20250109 21:08:02.934864 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.931082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:44567 (local address 127.12.126.65:59771)
0109 21:08:02.931351 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:02.931367 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:02.931480 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:02.932388 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:02.932393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:02.932415 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:02.932619 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.932626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.933635 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.933640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:02.934384 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.934396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.934507 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.934571 (+ 64us) client_negotiation.cc:769] Sending connection context
0109 21:08:02.934626 (+ 55us) client_negotiation.cc:240] Negotiation successful
0109 21:08:02.934693 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":83,"threads_started":1}
I20250109 21:08:02.935516 29068 leader_election.cc:290] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d56ac4f18fc842ec9977c65395db3206 (127.12.126.67:43475), b3f718ba5aab462a8a89ad217d05c03e (127.12.126.65:39075)
I20250109 21:08:02.935968 29216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.931692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:59771 (local address 127.12.126.66:44567)
0109 21:08:02.931971 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:02.931978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:02.931995 (+ 17us) server_negotiation.cc:407] Connection header received
0109 21:08:02.932052 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:02.932055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:02.932123 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:02.932253 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:02.932769 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.933534 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.935088 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.935552 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.935691 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:02.935751 (+ 60us) server_negotiation.cc:299] Negotiation successful
0109 21:08:02.935818 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
I20250109 21:08:02.936089 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.931958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:43475 (local address 127.12.126.65:39499)
0109 21:08:02.932784 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:02.932799 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:02.932897 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:02.933344 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:02.933348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:02.933370 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:02.933636 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.933646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.934680 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.934683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:02.935684 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.935695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.935807 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.935827 (+ 20us) client_negotiation.cc:769] Sending connection context
0109 21:08:02.935880 (+ 53us) client_negotiation.cc:240] Negotiation successful
0109 21:08:02.935937 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":76,"threads_started":1}
I20250109 21:08:02.936586 29209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.931658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:39499 (local address 127.12.126.67:43475)
0109 21:08:02.931817 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:02.931824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:02.932887 (+ 1063us) server_negotiation.cc:407] Connection header received
0109 21:08:02.933045 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:02.933051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:02.933134 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:02.933190 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:02.933789 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.934538 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.935819 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.936326 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.936358 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:02.936408 (+ 50us) server_negotiation.cc:299] Negotiation successful
0109 21:08:02.936465 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20250109 21:08:02.936812 29018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ebafb426bc42a7932289548e95c04b" candidate_uuid: "b3f718ba5aab462a8a89ad217d05c03e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63a58d7214904fd9b45c2f1cf75a3680" is_pre_election: true
I20250109 21:08:02.937028 29018 raft_consensus.cc:2463] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3f718ba5aab462a8a89ad217d05c03e in term 0.
I20250109 21:08:02.937070 29154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ebafb426bc42a7932289548e95c04b" candidate_uuid: "b3f718ba5aab462a8a89ad217d05c03e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d56ac4f18fc842ec9977c65395db3206" is_pre_election: true
I20250109 21:08:02.937328 29154 raft_consensus.cc:2463] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3f718ba5aab462a8a89ad217d05c03e in term 0.
I20250109 21:08:02.937448 28813 leader_election.cc:304] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63a58d7214904fd9b45c2f1cf75a3680, b3f718ba5aab462a8a89ad217d05c03e; no voters:
I20250109 21:08:02.937793 28931 raft_consensus.cc:2798] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:08:02.937873 28931 raft_consensus.cc:491] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:08:02.937920 28931 raft_consensus.cc:3054] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:02.939286 28931 raft_consensus.cc:513] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } }
I20250109 21:08:02.939651 28931 leader_election.cc:290] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 1 election: Requested vote from peers d56ac4f18fc842ec9977c65395db3206 (127.12.126.67:43475), 63a58d7214904fd9b45c2f1cf75a3680 (127.12.126.66:44567)
I20250109 21:08:02.939782 29219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.936157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:39075 (local address 127.12.126.66:45523)
0109 21:08:02.937556 (+ 1399us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:02.937572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:02.937669 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:02.937928 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:02.937931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:02.937951 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:02.938143 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.938149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.938845 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.938848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:02.939393 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.939401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.939508 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.939533 (+ 25us) client_negotiation.cc:769] Sending connection context
0109 21:08:02.939586 (+ 53us) client_negotiation.cc:240] Negotiation successful
0109 21:08:02.939645 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1326,"thread_start_us":85,"threads_started":1}
I20250109 21:08:02.940100 29154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ebafb426bc42a7932289548e95c04b" candidate_uuid: "b3f718ba5aab462a8a89ad217d05c03e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d56ac4f18fc842ec9977c65395db3206"
I20250109 21:08:02.940234 29154 raft_consensus.cc:3054] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:02.940558 29018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ebafb426bc42a7932289548e95c04b" candidate_uuid: "b3f718ba5aab462a8a89ad217d05c03e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63a58d7214904fd9b45c2f1cf75a3680"
I20250109 21:08:02.940665 29018 raft_consensus.cc:3054] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:02.941098 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.935806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:43475 (local address 127.12.126.66:58775)
0109 21:08:02.937719 (+ 1913us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:02.937735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:02.937868 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:02.938245 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:02.938248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:02.938271 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:02.938499 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.938509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.939446 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.939449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:02.940286 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:02.940298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.940811 (+ 513us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.940824 (+ 13us) client_negotiation.cc:769] Sending connection context
0109 21:08:02.940908 (+ 84us) client_negotiation.cc:240] Negotiation successful
0109 21:08:02.940968 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"thread_start_us":57,"threads_started":1}
I20250109 21:08:02.941112 29209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.936474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:58775 (local address 127.12.126.67:43475)
0109 21:08:02.936747 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:02.936753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:02.937797 (+ 1044us) server_negotiation.cc:407] Connection header received
0109 21:08:02.938010 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:02.938014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:02.938063 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:02.938113 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:02.938630 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.939301 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.940420 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.940733 (+ 313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.940761 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:02.940945 (+ 184us) server_negotiation.cc:299] Negotiation successful
0109 21:08:02.941003 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":118,"threads_started":1}
I20250109 21:08:02.941493 29154 raft_consensus.cc:2463] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3f718ba5aab462a8a89ad217d05c03e in term 1.
I20250109 21:08:02.941803 29018 raft_consensus.cc:2463] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3f718ba5aab462a8a89ad217d05c03e in term 1.
I20250109 21:08:02.941857 28812 leader_election.cc:304] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3f718ba5aab462a8a89ad217d05c03e, d56ac4f18fc842ec9977c65395db3206; no voters:
I20250109 21:08:02.942034 28931 raft_consensus.cc:2798] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:02.942216 28931 raft_consensus.cc:695] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 LEADER]: Becoming Leader. State: Replica: b3f718ba5aab462a8a89ad217d05c03e, State: Running, Role: LEADER
I20250109 21:08:02.942287 29154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ebafb426bc42a7932289548e95c04b" candidate_uuid: "63a58d7214904fd9b45c2f1cf75a3680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d56ac4f18fc842ec9977c65395db3206" is_pre_election: true
I20250109 21:08:02.942445 29154 raft_consensus.cc:2388] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63a58d7214904fd9b45c2f1cf75a3680 in current term 1: Already voted for candidate b3f718ba5aab462a8a89ad217d05c03e in this term.
I20250109 21:08:02.942284 29220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:02.936518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:45523 (local address 127.12.126.65:39075)
0109 21:08:02.936865 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:02.936870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:02.937750 (+ 880us) server_negotiation.cc:407] Connection header received
0109 21:08:02.937789 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:02.937791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:02.937833 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:02.937906 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:02.938259 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.938712 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:02.939506 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:02.940131 (+ 625us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:02.940243 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:02.940297 (+ 54us) server_negotiation.cc:299] Negotiation successful
0109 21:08:02.940364 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":64,"threads_started":1}
I20250109 21:08:02.942490 28931 consensus_queue.cc:237] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } }
I20250109 21:08:02.943007 28879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ebafb426bc42a7932289548e95c04b" candidate_uuid: "63a58d7214904fd9b45c2f1cf75a3680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3f718ba5aab462a8a89ad217d05c03e" is_pre_election: true
I20250109 21:08:02.943603 28950 leader_election.cc:304] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63a58d7214904fd9b45c2f1cf75a3680; no voters: b3f718ba5aab462a8a89ad217d05c03e, d56ac4f18fc842ec9977c65395db3206
I20250109 21:08:02.943831 29068 raft_consensus.cc:2743] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 21:08:02.945750 28736 catalog_manager.cc:5526] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e reported cstate change: term changed from 0 to 1, leader changed from <none> to b3f718ba5aab462a8a89ad217d05c03e (127.12.126.65). New cstate: current_term: 1 leader_uuid: "b3f718ba5aab462a8a89ad217d05c03e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } health_report { overall_health: HEALTHY } } }
I20250109 21:08:03.024854 29220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:03.021610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.12.126.65:39075)
0109 21:08:03.021765 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:03.021771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:03.021786 (+ 15us) server_negotiation.cc:407] Connection header received
0109 21:08:03.021971 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:03.021973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:03.022034 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:03.022145 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:08:03.022491 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:03.023113 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:03.023739 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:03.023932 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:03.023993 (+ 61us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:08:03.024439 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:08:03.024548 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:03.024638 (+ 90us) server_negotiation.cc:299] Negotiation successful
0109 21:08:03.024695 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250109 21:08:03.028730 29154 raft_consensus.cc:1270] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206 [term 1 FOLLOWER]: Refusing update from remote peer b3f718ba5aab462a8a89ad217d05c03e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:08:03.028759 29018 raft_consensus.cc:1270] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680 [term 1 FOLLOWER]: Refusing update from remote peer b3f718ba5aab462a8a89ad217d05c03e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:08:03.029368 29222 consensus_queue.cc:1035] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [LEADER]: Connected to new peer: Peer: permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:08:03.029546 28931 consensus_queue.cc:1035] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:08:03.035559 29225 mvcc.cc:204] Tried to move back new op lower bound from 7112527392879722496 to 7112527392535973888. Current Snapshot: MvccSnapshot[applied={T|T < 7112527392879722496}]
I20250109 21:08:03.037207 29226 mvcc.cc:204] Tried to move back new op lower bound from 7112527392879722496 to 7112527392535973888. Current Snapshot: MvccSnapshot[applied={T|T < 7112527392879722496}]
I20250109 21:08:03.040858 29227 mvcc.cc:204] Tried to move back new op lower bound from 7112527392879722496 to 7112527392535973888. Current Snapshot: MvccSnapshot[applied={T|T < 7112527392879722496 or (T in {7112527392879722496})}]
I20250109 21:08:03.043272 28737 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48588:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:08:03.043412 28737 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48588:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:08:03.045686 28737 catalog_manager.cc:5813] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: Sending DeleteTablet for 3 replicas of tablet d8ebafb426bc42a7932289548e95c04b
I20250109 21:08:03.046331 28998 tablet_service.cc:1514] Processing DeleteTablet for tablet d8ebafb426bc42a7932289548e95c04b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:03 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:42494
I20250109 21:08:03.046490 28859 tablet_service.cc:1514] Processing DeleteTablet for tablet d8ebafb426bc42a7932289548e95c04b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:03 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:50858
I20250109 21:08:03.046527 29134 tablet_service.cc:1514] Processing DeleteTablet for tablet d8ebafb426bc42a7932289548e95c04b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:03 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:49130
I20250109 21:08:03.046844 29236 tablet_replica.cc:331] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e: stopping tablet replica
I20250109 21:08:03.046959 29236 raft_consensus.cc:2238] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:08:03.047010 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 28775
I20250109 21:08:03.047156 29237 tablet_replica.cc:331] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206: stopping tablet replica
I20250109 21:08:03.047309 29236 raft_consensus.cc:2267] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:03.047413 29237 raft_consensus.cc:2238] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:08:03.047928 29237 raft_consensus.cc:2267] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:03.048884 29235 tablet_replica.cc:331] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680: stopping tablet replica
I20250109 21:08:03.049006 29235 raft_consensus.cc:2238] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:08:03.049137 29235 raft_consensus.cc:2267] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:03.049432 29237 ts_tablet_manager.cc:1905] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:08:03.049886 29235 ts_tablet_manager.cc:1905] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:08:03.051882 29235 ts_tablet_manager.cc:1918] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:08:03.051954 29235 log.cc:1198] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/d8ebafb426bc42a7932289548e95c04b
I20250109 21:08:03.052258 29235 ts_tablet_manager.cc:1939] T d8ebafb426bc42a7932289548e95c04b P 63a58d7214904fd9b45c2f1cf75a3680: Deleting consensus metadata
I20250109 21:08:03.052551 29237 ts_tablet_manager.cc:1918] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:08:03.052649 29237 log.cc:1198] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/d8ebafb426bc42a7932289548e95c04b
I20250109 21:08:03.052934 28723 catalog_manager.cc:4872] TS 63a58d7214904fd9b45c2f1cf75a3680 (127.12.126.66:44567): tablet d8ebafb426bc42a7932289548e95c04b (table test-table [id=af14e996f29647d79641360d4f97bc8a]) successfully deleted
I20250109 21:08:03.053030 29237 ts_tablet_manager.cc:1939] T d8ebafb426bc42a7932289548e95c04b P d56ac4f18fc842ec9977c65395db3206: Deleting consensus metadata
I20250109 21:08:03.054289 28722 catalog_manager.cc:4872] TS d56ac4f18fc842ec9977c65395db3206 (127.12.126.67:43475): tablet d8ebafb426bc42a7932289548e95c04b (table test-table [id=af14e996f29647d79641360d4f97bc8a]) successfully deleted
W20250109 21:08:03.055362 28721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.126.65:39075 (error 108)
I20250109 21:08:03.055737 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 28929
I20250109 21:08:03.056828 28932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:03.056455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:39075 (local address 127.0.0.1:50872)
0109 21:08:03.056664 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:03.056728 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.65:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20250109 21:08:03.056965 28721 catalog_manager.cc:4615] TS b3f718ba5aab462a8a89ad217d05c03e (127.12.126.65:39075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8ebafb426bc42a7932289548e95c04b: Network error: Client connection negotiation failed: client connection to 127.12.126.65:39075: connect: Connection refused (error 111)
I20250109 21:08:03.062067 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 29069
I20250109 21:08:03.067852 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 28704
I20250109 21:08:23.075093 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:35275
--webserver_interface=127.12.126.126
--webserver_port=46205
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:35275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:23.179399 29239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:23.179647 29239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:23.179716 29239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:23.182888 29239 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20250109 21:08:23.182942 29239 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20250109 21:08:23.182960 29239 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20250109 21:08:23.182994 29239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:08:23.183038 29239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:23.183075 29239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:23.183121 29239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:08:23.183159 29239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:08:23.187516 29239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:35275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:35275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=46205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:23.188570 29239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:23.189404 29239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:23.196321 29247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:23.197044 29244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:23.197820 29239 server_base.cc:1034] running on GCE node
W20250109 21:08:23.198618 29245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:23.199151 29239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:23.200044 29239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:23.201195 29239 hybrid_clock.cc:648] HybridClock initialized: now 1736456903201182 us; error 25 us; skew 500 ppm
Jan 09 21:08:23 dist-test-slave-g0zb krb5kdc[28156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456903, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:23.203861 29239 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:08:23.204726 29239 webserver.cc:458] Webserver started at http://127.12.126.126:46205/ using document root <none> and password file <none>
I20250109 21:08:23.205278 29239 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:23.205338 29239 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:23.208685 29239 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:08:23.210474 29254 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:23.211447 29239 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:23.211552 29239 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "cf5434de2d4448068da2c603fb583b04"
format_stamp: "Formatted at 2025-01-09 21:07:40 on dist-test-slave-g0zb"
server_key: "33320c6b3e8aa2962f9f4bfff1b9b7b3"
server_key_iv: "74abe7bf2c0fffc09886a847e60e63a3"
server_key_version: "encryptionkey@0"
I20250109 21:08:23.211946 29239 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:23.230983 29239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:23.233680 29239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:23.233844 29239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:23.240334 29239 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:35275
I20250109 21:08:23.240556 29306 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:35275 every 8 connection(s)
I20250109 21:08:23.241380 29239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:08:23.245206 29307 sys_catalog.cc:263] Verifying existing consensus state
I20250109 21:08:23.246711 29307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: Bootstrap starting.
I20250109 21:08:23.251513 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 29239
I20250109 21:08:23.251927 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:39075
--local_ip_for_outbound_sockets=127.12.126.65
--tserver_master_addrs=127.12.126.126:35275
--webserver_port=41015
--webserver_interface=127.12.126.65
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:08:23.263495 29307 log.cc:826] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:23.274127 29307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:08:23.274498 29307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: Bootstrap complete.
I20250109 21:08:23.277976 29307 raft_consensus.cc:357] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5434de2d4448068da2c603fb583b04" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35275 } }
I20250109 21:08:23.278345 29307 raft_consensus.cc:738] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5434de2d4448068da2c603fb583b04, State: Initialized, Role: FOLLOWER
I20250109 21:08:23.278927 29307 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5434de2d4448068da2c603fb583b04" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35275 } }
I20250109 21:08:23.279104 29307 raft_consensus.cc:397] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:08:23.279177 29307 raft_consensus.cc:491] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:08:23.279285 29307 raft_consensus.cc:3054] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 2 FOLLOWER]: Advancing to term 3
I20250109 21:08:23.280812 29307 raft_consensus.cc:513] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5434de2d4448068da2c603fb583b04" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35275 } }
I20250109 21:08:23.281131 29307 leader_election.cc:304] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf5434de2d4448068da2c603fb583b04; no voters:
I20250109 21:08:23.281406 29307 leader_election.cc:290] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250109 21:08:23.281498 29311 raft_consensus.cc:2798] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 3 FOLLOWER]: Leader election won for term 3
I20250109 21:08:23.281684 29311 raft_consensus.cc:695] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [term 3 LEADER]: Becoming Leader. State: Replica: cf5434de2d4448068da2c603fb583b04, State: Running, Role: LEADER
I20250109 21:08:23.282053 29311 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5434de2d4448068da2c603fb583b04" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35275 } }
I20250109 21:08:23.282536 29307 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:08:23.283449 29313 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf5434de2d4448068da2c603fb583b04. Latest consensus state: current_term: 3 leader_uuid: "cf5434de2d4448068da2c603fb583b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5434de2d4448068da2c603fb583b04" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35275 } } }
I20250109 21:08:23.283490 29312 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cf5434de2d4448068da2c603fb583b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5434de2d4448068da2c603fb583b04" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35275 } } }
I20250109 21:08:23.283608 29312 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:23.283560 29313 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:23.284024 29319 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:08:23.288899 29319 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:08:23.289674 29319 catalog_manager.cc:1260] Loaded cluster ID: 2f406a50bda84df8bbf80dc3ab5f3401
I20250109 21:08:23.289728 29319 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:08:23.291512 29319 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:08:23.292105 29319 catalog_manager.cc:5910] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: Loaded TSK: 0
I20250109 21:08:23.292174 29319 catalog_manager.cc:5910] T 00000000000000000000000000000000 P cf5434de2d4448068da2c603fb583b04: Loaded TSK: 1
F20250109 21:08:23.292677 29319 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa69636ddcd google::LogMessage::Fail() at ??:0
@ 0x7fa696371b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa69636d7cc google::LogMessage::Flush() at ??:0
@ 0x7fa69636ef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa6943b51f0 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa6943ad8e0 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa69b2a8799 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa69b280e4d kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa69b2816df _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fa69b2c693c _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fa69b2e65b2 std::function<>::operator()() at ??:0
@ 0x7fa69b281026 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fa69b282560 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa69b27e84b _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fa69b2c5dd9 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x560d93559bbe std::function<>::operator()() at ??:0
@ 0x7fa6977d8a48 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa6977d9331 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa6977dac7d _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x560d93559bbe std::function<>::operator()() at ??:0
@ 0x7fa6977c8495 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa6980106db start_thread at ??:0
@ 0x7fa69552471f clone at ??:0
W20250109 21:08:23.397843 29309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:23.398069 29309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:23.398138 29309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:23.401304 29309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:23.401358 29309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:23.401461 29309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:08:23.405638 29309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:39075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=41015
--tserver_master_addrs=127.12.126.126:35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:23.406715 29309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:23.407621 29309 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:23.415289 29355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:23.416071 29358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:23.416687 29356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:23.417394 29309 server_base.cc:1034] running on GCE node
I20250109 21:08:23.417856 29309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:23.418589 29309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:23.419770 29309 hybrid_clock.cc:648] HybridClock initialized: now 1736456903419736 us; error 44 us; skew 500 ppm
Jan 09 21:08:23 dist-test-slave-g0zb krb5kdc[28156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456903, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:23.423173 29309 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:08:23.424371 29309 webserver.cc:458] Webserver started at http://127.12.126.65:41015/ using document root <none> and password file <none>
I20250109 21:08:23.425110 29309 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:23.425199 29309 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:23.429733 29309 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:08:23.432173 29365 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:23.433396 29309 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:08:23.433516 29309 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "b3f718ba5aab462a8a89ad217d05c03e"
format_stamp: "Formatted at 2025-01-09 21:07:41 on dist-test-slave-g0zb"
server_key: "4d4f042dc7676813ace540dccc9c62c7"
server_key_iv: "b7780acea10b144c883f2d5003e98ba6"
server_key_version: "encryptionkey@0"
I20250109 21:08:23.434093 29309 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:23.445528 29309 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:23.448483 29309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:23.448637 29309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:23.449126 29309 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:23.450120 29372 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250109 21:08:23.455010 29309 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250109 21:08:23.455077 29309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20250109 21:08:23.455164 29309 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250109 21:08:23.456702 29309 ts_tablet_manager.cc:610] Registered 1 tablets
I20250109 21:08:23.456744 29309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20250109 21:08:23.457036 29372 tablet_bootstrap.cc:492] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e: Bootstrap starting.
I20250109 21:08:23.466970 29309 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:39075
I20250109 21:08:23.467249 29479 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:39075 every 8 connection(s)
I20250109 21:08:23.468061 29309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:08:23.478219 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 29309
I20250109 21:08:23.478570 29372 log.cc:826] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:23.478678 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:44567
--local_ip_for_outbound_sockets=127.12.126.66
--tserver_master_addrs=127.12.126.126:35275
--webserver_port=41565
--webserver_interface=127.12.126.66
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:08:23.486893 29372 tablet_bootstrap.cc:492] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 21:08:23.487316 29372 tablet_bootstrap.cc:492] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e: Bootstrap complete.
I20250109 21:08:23.488677 29372 ts_tablet_manager.cc:1397] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.004s
I20250109 21:08:23.491499 29372 raft_consensus.cc:357] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } }
I20250109 21:08:23.491940 29372 raft_consensus.cc:738] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f718ba5aab462a8a89ad217d05c03e, State: Initialized, Role: FOLLOWER
I20250109 21:08:23.492460 29372 consensus_queue.cc:260] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } }
I20250109 21:08:23.493353 29372 ts_tablet_manager.cc:1428] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20250109 21:08:23.522549 29482 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:08:23.469839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:58159)
0109 21:08:23.470294 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:23.470335 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:23.471082 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:23.521813 (+ 50731us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":106,"threads_started":1}
W20250109 21:08:23.523061 29480 heartbeater.cc:643] Failed to heartbeat to 127.12.126.126:35275 (0 consecutive failures): Network error: Failed to ping master at 127.12.126.126:35275: Client connection negotiation failed: client connection to 127.12.126.126:35275: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250109 21:08:23.523936 29482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:23.523572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:42835)
0109 21:08:23.523754 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:23.523836 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20250109 21:08:23.524626 29482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:23.524380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:41861)
0109 21:08:23.524512 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:23.524568 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20250109 21:08:23.524791 29480 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250109 21:08:23.589078 29484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:23.589323 29484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:23.589378 29484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:23.592396 29484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:23.592458 29484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:23.592537 29484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:08:23.596683 29484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:44567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=41565
--tserver_master_addrs=127.12.126.126:35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:23.597734 29484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:23.598593 29484 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:23.606251 29490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:23.606990 29491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:23.607570 29493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:23.608328 29484 server_base.cc:1034] running on GCE node
I20250109 21:08:23.608692 29484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:23.609241 29484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:23.610383 29484 hybrid_clock.cc:648] HybridClock initialized: now 1736456903610363 us; error 32 us; skew 500 ppm
Jan 09 21:08:23 dist-test-slave-g0zb krb5kdc[28156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456903, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:23.618409 29484 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:08:23.619382 29484 webserver.cc:458] Webserver started at http://127.12.126.66:41565/ using document root <none> and password file <none>
I20250109 21:08:23.619927 29484 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:23.619983 29484 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:23.623379 29484 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:08:23.625167 29500 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:23.626139 29484 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:23.626241 29484 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "63a58d7214904fd9b45c2f1cf75a3680"
format_stamp: "Formatted at 2025-01-09 21:07:41 on dist-test-slave-g0zb"
server_key: "ea721f122ba6c1950019514e820fc7ca"
server_key_iv: "223efb89a09aae1d4c5ca6e020dd40c3"
server_key_version: "encryptionkey@0"
I20250109 21:08:23.626667 29484 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:23.643507 29484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:23.646046 29484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:23.646209 29484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:23.646696 29484 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:23.647467 29484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:23.647516 29484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:23.647578 29484 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:23.647604 29484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:23.655730 29484 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:44567
I20250109 21:08:23.655759 29613 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:44567 every 8 connection(s)
I20250109 21:08:23.656700 29484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:08:23.659832 29616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:23.658495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:60687)
0109 21:08:23.658909 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:23.658998 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":86,"threads_started":1}
W20250109 21:08:23.660216 29614 heartbeater.cc:643] Failed to heartbeat to 127.12.126.126:35275 (0 consecutive failures): Network error: Failed to ping master at 127.12.126.126:35275: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
I20250109 21:08:23.660704 29616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:23.660518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:58605)
0109 21:08:23.660612 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:23.660653 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20250109 21:08:23.661216 29616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:23.661067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:55025)
0109 21:08:23.661143 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:23.661177 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20250109 21:08:23.661360 29614 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20250109 21:08:23.666790 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 29484
I20250109 21:08:23.667254 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:43475
--local_ip_for_outbound_sockets=127.12.126.67
--tserver_master_addrs=127.12.126.126:35275
--webserver_port=45797
--webserver_interface=127.12.126.67
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:23.773887 29617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:23.774153 29617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:23.774233 29617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:23.777321 29617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:23.777377 29617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:23.777477 29617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:08:23.781517 29617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:43475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=45797
--tserver_master_addrs=127.12.126.126:35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:23.782522 29617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:23.783404 29617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:23.791054 29625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:23.791754 29623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:23.792304 29622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:23.793035 29617 server_base.cc:1034] running on GCE node
I20250109 21:08:23.793370 29617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:23.793893 29617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:23.795058 29617 hybrid_clock.cc:648] HybridClock initialized: now 1736456903795045 us; error 27 us; skew 500 ppm
Jan 09 21:08:23 dist-test-slave-g0zb krb5kdc[28156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456903, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:23.797531 29617 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:08:23.798363 29617 webserver.cc:458] Webserver started at http://127.12.126.67:45797/ using document root <none> and password file <none>
I20250109 21:08:23.798899 29617 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:23.798954 29617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:23.802238 29617 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:08:23.804128 29632 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:23.805024 29617 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:08:23.805125 29617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "d56ac4f18fc842ec9977c65395db3206"
format_stamp: "Formatted at 2025-01-09 21:07:41 on dist-test-slave-g0zb"
server_key: "c8481caa2dd1d37ffbe0d51e88690ae3"
server_key_iv: "5f7399b947afe57bc68d9e89c9cbf262"
server_key_version: "encryptionkey@0"
I20250109 21:08:23.805549 29617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:23.855906 29617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:23.858917 29617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:23.859057 29617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:23.859560 29617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:23.860389 29617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:23.860433 29617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:23.860481 29617 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:23.860512 29617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:23.868760 29617 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:43475
I20250109 21:08:23.868853 29745 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:43475 every 8 connection(s)
I20250109 21:08:23.869793 29617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:08:23.873135 29748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:23.871618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:46293)
0109 21:08:23.872023 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:23.872124 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":92,"threads_started":1}
W20250109 21:08:23.873543 29746 heartbeater.cc:643] Failed to heartbeat to 127.12.126.126:35275 (0 consecutive failures): Network error: Failed to ping master at 127.12.126.126:35275: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
I20250109 21:08:23.873766 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 29617
I20250109 21:08:23.874205 29748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:23.873977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:46599)
0109 21:08:23.874084 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:23.874129 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20250109 21:08:23.875087 29748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:23.874894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:34019)
0109 21:08:23.874987 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:23.875028 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20250109 21:08:23.875237 29746 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20250109 21:08:24.525763 29750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:24.525370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:33827)
0109 21:08:24.525611 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:24.525664 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":78,"threads_started":1}
I20250109 21:08:24.638427 29751 raft_consensus.cc:491] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:08:24.638628 29751 raft_consensus.cc:513] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } }
I20250109 21:08:24.639722 29751 leader_election.cc:290] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d56ac4f18fc842ec9977c65395db3206 (127.12.126.67:43475), 63a58d7214904fd9b45c2f1cf75a3680 (127.12.126.66:44567)
Jan 09 21:08:24 dist-test-slave-g0zb krb5kdc[28156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456903, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.66@KRBTEST.COM
Jan 09 21:08:24 dist-test-slave-g0zb krb5kdc[28156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456903, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.67@KRBTEST.COM
I20250109 21:08:24.652784 29750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:24.639927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:44567 (local address 127.12.126.65:39459)
0109 21:08:24.640144 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:24.640162 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:24.640266 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:24.642086 (+ 1820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:24.642092 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:24.642721 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:24.643501 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:24.643519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:24.645085 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:24.645088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:24.645681 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:24.645686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:24.645851 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:24.646532 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:08:24.646559 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:24.648191 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:24.650747 (+ 2556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:24.650751 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:24.650759 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:24.651019 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:24.651275 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:24.651277 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:24.651279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:24.651357 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:24.651900 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:24.651906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:24.652106 (+ 200us) client_negotiation.cc:769] Sending connection context
0109 21:08:24.652272 (+ 166us) client_negotiation.cc:240] Negotiation successful
0109 21:08:24.652525 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"spinlock_wait_cycles":9728}
I20250109 21:08:24.652799 29753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:24.640080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:39459 (local address 127.12.126.66:44567)
0109 21:08:24.640446 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:24.640459 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:24.640500 (+ 41us) server_negotiation.cc:407] Connection header received
0109 21:08:24.641217 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:24.641225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:24.641619 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:24.642015 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:24.643672 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:24.644955 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:24.645775 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:24.646046 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:24.648362 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:24.648425 (+ 63us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:24.648441 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:24.648485 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:24.650624 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:24.651109 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:24.651114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:24.651118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:24.651182 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:24.651443 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:24.651445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:24.651446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:24.651785 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:24.651912 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:24.652407 (+ 495us) server_negotiation.cc:299] Negotiation successful
0109 21:08:24.652625 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":71,"threads_started":1}
I20250109 21:08:24.652807 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:24.640111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:37393 (local address 127.12.126.67:43475)
0109 21:08:24.640446 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:24.640459 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:24.640500 (+ 41us) server_negotiation.cc:407] Connection header received
0109 21:08:24.641217 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:24.641225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:24.641619 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:24.642015 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:24.643672 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:24.644955 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:24.645775 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:24.646046 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:24.648671 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:24.648730 (+ 59us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:24.648745 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:24.648785 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:24.650639 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:24.651339 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:24.651343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:24.651346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:24.651409 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:24.651620 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:24.651623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:24.651625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:24.651941 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:24.652068 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:24.652407 (+ 339us) server_negotiation.cc:299] Negotiation successful
0109 21:08:24.652615 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":73,"threads_started":1}
I20250109 21:08:24.652832 29752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:24.639927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:43475 (local address 127.12.126.65:37393)
0109 21:08:24.640255 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:24.640267 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:24.640331 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:24.642086 (+ 1755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:24.642092 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:24.642721 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:24.643501 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:24.643519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:24.645078 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:24.645081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:24.645664 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:24.645676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:24.645851 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:24.646730 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:08:24.646743 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:24.648519 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:24.651109 (+ 2590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:24.651111 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:24.651112 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:24.651238 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:24.651494 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:24.651497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:24.651499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:24.651540 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:24.652046 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:24.652049 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:24.652135 (+ 86us) client_negotiation.cc:769] Sending connection context
0109 21:08:24.652271 (+ 136us) client_negotiation.cc:240] Negotiation successful
0109 21:08:24.652615 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":161,"spinlock_wait_cycles":10752,"thread_start_us":57,"threads_started":1}
W20250109 21:08:24.655203 29367 leader_election.cc:343] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d56ac4f18fc842ec9977c65395db3206 (127.12.126.67:43475): Not found: Tablet not found: d8ebafb426bc42a7932289548e95c04b
W20250109 21:08:24.655416 29368 leader_election.cc:343] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63a58d7214904fd9b45c2f1cf75a3680 (127.12.126.66:44567): Not found: Tablet not found: d8ebafb426bc42a7932289548e95c04b
I20250109 21:08:24.655478 29368 leader_election.cc:304] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3f718ba5aab462a8a89ad217d05c03e; no voters: 63a58d7214904fd9b45c2f1cf75a3680, d56ac4f18fc842ec9977c65395db3206
I20250109 21:08:24.655709 29751 raft_consensus.cc:2743] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 21:08:24.662137 29755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:24.661760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:42323)
0109 21:08:24.661998 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:24.662055 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":95,"threads_started":1}
I20250109 21:08:24.876048 29756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:24.875700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:47439)
0109 21:08:24.875924 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:24.875969 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20250109 21:08:25.527165 29757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:25.526636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:44597)
0109 21:08:25.526961 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:25.527036 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":97,"threads_started":1}
I20250109 21:08:25.663460 29758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:25.662981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:56053)
0109 21:08:25.663271 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:25.663348 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
I20250109 21:08:25.877373 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:25.876930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:33853)
0109 21:08:25.877197 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:25.877259 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20250109 21:08:26.484809 29760 raft_consensus.cc:491] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:08:26.484936 29760 raft_consensus.cc:513] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } }
I20250109 21:08:26.485359 29760 leader_election.cc:290] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d56ac4f18fc842ec9977c65395db3206 (127.12.126.67:43475), 63a58d7214904fd9b45c2f1cf75a3680 (127.12.126.66:44567)
W20250109 21:08:26.486166 29367 leader_election.cc:343] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d56ac4f18fc842ec9977c65395db3206 (127.12.126.67:43475): Not found: Tablet not found: d8ebafb426bc42a7932289548e95c04b
W20250109 21:08:26.486246 29368 leader_election.cc:343] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63a58d7214904fd9b45c2f1cf75a3680 (127.12.126.66:44567): Not found: Tablet not found: d8ebafb426bc42a7932289548e95c04b
I20250109 21:08:26.486348 29368 leader_election.cc:304] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3f718ba5aab462a8a89ad217d05c03e; no voters: 63a58d7214904fd9b45c2f1cf75a3680, d56ac4f18fc842ec9977c65395db3206
I20250109 21:08:26.486481 29760 raft_consensus.cc:2743] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 21:08:26.528276 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:26.527818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:56845)
0109 21:08:26.528096 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:26.528148 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
I20250109 21:08:26.664419 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:26.664069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:48419)
0109 21:08:26.664291 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:26.664340 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
I20250109 21:08:26.878576 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:26.878200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:33593)
0109 21:08:26.878440 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:26.878488 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":89,"threads_started":1}
I20250109 21:08:27.529302 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:27.528914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:43177)
0109 21:08:27.529151 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:27.529195 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
I20250109 21:08:27.665488 29765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:27.665130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:55963)
0109 21:08:27.665355 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:27.665403 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20250109 21:08:27.879743 29766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:27.879363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:46233)
0109 21:08:27.879600 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:27.879652 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
I20250109 21:08:28.530766 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:28.530200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:40765)
0109 21:08:28.530587 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:28.530651 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":76,"threads_started":1}
I20250109 21:08:28.666556 29768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:28.666168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:57385)
0109 21:08:28.666410 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:28.666462 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20250109 21:08:28.785877 29769 raft_consensus.cc:491] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:08:28.786013 29769 raft_consensus.cc:513] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } }
I20250109 21:08:28.786441 29769 leader_election.cc:290] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d56ac4f18fc842ec9977c65395db3206 (127.12.126.67:43475), 63a58d7214904fd9b45c2f1cf75a3680 (127.12.126.66:44567)
W20250109 21:08:28.787248 29367 leader_election.cc:343] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d56ac4f18fc842ec9977c65395db3206 (127.12.126.67:43475): Not found: Tablet not found: d8ebafb426bc42a7932289548e95c04b
W20250109 21:08:28.787395 29368 leader_election.cc:343] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63a58d7214904fd9b45c2f1cf75a3680 (127.12.126.66:44567): Not found: Tablet not found: d8ebafb426bc42a7932289548e95c04b
I20250109 21:08:28.787482 29368 leader_election.cc:304] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3f718ba5aab462a8a89ad217d05c03e; no voters: 63a58d7214904fd9b45c2f1cf75a3680, d56ac4f18fc842ec9977c65395db3206
I20250109 21:08:28.787642 29769 raft_consensus.cc:2743] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 21:08:28.880856 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:28.880501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:39581)
0109 21:08:28.880715 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:28.880763 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":71,"threads_started":1}
I20250109 21:08:29.531891 29771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:29.531504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:33857)
0109 21:08:29.531750 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:29.531798 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":91,"threads_started":1}
I20250109 21:08:29.667654 29772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:29.667237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:54123)
0109 21:08:29.667490 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:29.667554 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":69,"threads_started":1}
I20250109 21:08:29.882091 29773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:29.881588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:59497)
0109 21:08:29.881897 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:29.881959 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":86,"threads_started":1}
I20250109 21:08:30.532930 29774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:30.532541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:47069)
0109 21:08:30.532779 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:30.532836 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20250109 21:08:30.668851 29775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:30.668410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:46429)
0109 21:08:30.668690 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:30.668744 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":69,"threads_started":1}
I20250109 21:08:30.883443 29776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:30.882970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:57343)
0109 21:08:30.883294 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:30.883350 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":103,"threads_started":1}
I20250109 21:08:31.395778 29777 raft_consensus.cc:491] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:08:31.395908 29777 raft_consensus.cc:513] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } }
I20250109 21:08:31.396353 29777 leader_election.cc:290] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d56ac4f18fc842ec9977c65395db3206 (127.12.126.67:43475), 63a58d7214904fd9b45c2f1cf75a3680 (127.12.126.66:44567)
W20250109 21:08:31.397046 29367 leader_election.cc:343] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d56ac4f18fc842ec9977c65395db3206 (127.12.126.67:43475): Not found: Tablet not found: d8ebafb426bc42a7932289548e95c04b
W20250109 21:08:31.397132 29368 leader_election.cc:343] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63a58d7214904fd9b45c2f1cf75a3680 (127.12.126.66:44567): Not found: Tablet not found: d8ebafb426bc42a7932289548e95c04b
I20250109 21:08:31.397229 29368 leader_election.cc:304] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3f718ba5aab462a8a89ad217d05c03e; no voters: 63a58d7214904fd9b45c2f1cf75a3680, d56ac4f18fc842ec9977c65395db3206
I20250109 21:08:31.397378 29777 raft_consensus.cc:2743] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 21:08:31.534122 29778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:31.533732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:47379)
0109 21:08:31.533977 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:31.534025 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":71,"threads_started":1}
I20250109 21:08:31.670130 29779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:31.669716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:51219)
0109 21:08:31.669968 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:31.670018 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20250109 21:08:31.884362 29780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:31.884002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:50739)
0109 21:08:31.884226 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:31.884273 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":65,"threads_started":1}
I20250109 21:08:32.535387 29781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:32.534975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:50147)
0109 21:08:32.535223 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:32.535291 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20250109 21:08:32.671131 29782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:32.670743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:43499)
0109 21:08:32.670997 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:32.671048 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":93,"threads_started":1}
I20250109 21:08:32.885447 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:32.885075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:59545)
0109 21:08:32.885309 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:32.885357 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20250109 21:08:33.536505 29784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:33.536103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:52061)
0109 21:08:33.536360 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:33.536410 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
I20250109 21:08:33.672076 29785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:33.671734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:48069)
0109 21:08:33.671952 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:33.671996 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":71,"threads_started":1}
I20250109 21:08:33.886618 29786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:33.886133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:34235)
0109 21:08:33.886473 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:33.886528 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":70,"threads_started":1}
I20250109 21:08:34.537569 29787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:34.537173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:40173)
0109 21:08:34.537420 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:34.537470 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":90,"threads_started":1}
I20250109 21:08:34.673157 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:34.672753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:55777)
0109 21:08:34.673003 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:34.673056 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
I20250109 21:08:34.887756 29789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:34.887344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:55573)
0109 21:08:34.887595 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:34.887650 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
I20250109 21:08:34.891893 29790 raft_consensus.cc:491] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:08:34.892015 29790 raft_consensus.cc:513] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56ac4f18fc842ec9977c65395db3206" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43475 } } peers { permanent_uuid: "63a58d7214904fd9b45c2f1cf75a3680" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 44567 } } peers { permanent_uuid: "b3f718ba5aab462a8a89ad217d05c03e" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39075 } }
I20250109 21:08:34.892396 29790 leader_election.cc:290] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d56ac4f18fc842ec9977c65395db3206 (127.12.126.67:43475), 63a58d7214904fd9b45c2f1cf75a3680 (127.12.126.66:44567)
W20250109 21:08:34.893172 29367 leader_election.cc:343] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d56ac4f18fc842ec9977c65395db3206 (127.12.126.67:43475): Not found: Tablet not found: d8ebafb426bc42a7932289548e95c04b
W20250109 21:08:34.893273 29368 leader_election.cc:343] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63a58d7214904fd9b45c2f1cf75a3680 (127.12.126.66:44567): Not found: Tablet not found: d8ebafb426bc42a7932289548e95c04b
I20250109 21:08:34.893362 29368 leader_election.cc:304] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3f718ba5aab462a8a89ad217d05c03e; no voters: 63a58d7214904fd9b45c2f1cf75a3680, d56ac4f18fc842ec9977c65395db3206
I20250109 21:08:34.893524 29790 raft_consensus.cc:2743] T d8ebafb426bc42a7932289548e95c04b P b3f718ba5aab462a8a89ad217d05c03e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 21:08:35.538684 29791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:35.538235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:49799)
0109 21:08:35.538513 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:35.538566 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":83,"threads_started":1}
I20250109 21:08:35.674345 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:35.673954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:53239)
0109 21:08:35.674196 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:35.674252 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":87,"threads_started":1}
I20250109 21:08:35.888957 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:35.888473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:36467)
0109 21:08:35.888737 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:35.888817 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":104,"threads_started":1}
I20250109 21:08:36.539856 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:36.539428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:46983)
0109 21:08:36.539694 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:36.539746 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":92,"threads_started":1}
I20250109 21:08:36.675486 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:36.675131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:56295)
0109 21:08:36.675353 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:36.675402 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":71,"threads_started":1}
I20250109 21:08:36.890110 29796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:36.889674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:50609)
0109 21:08:36.889944 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:36.890002 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":88,"threads_started":1}
I20250109 21:08:37.541071 29797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:37.540627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:37323)
0109 21:08:37.540908 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:37.540970 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":85,"threads_started":1}
I20250109 21:08:37.676633 29798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:37.676226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:48745)
0109 21:08:37.676478 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:37.676534 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
I20250109 21:08:37.891281 29799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:37.890874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:50081)
0109 21:08:37.891132 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:37.891185 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
I20250109 21:08:38.542647 29800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:38.542129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.65:36093)
0109 21:08:38.542460 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:38.542549 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":95,"threads_started":1}
I20250109 21:08:38.678251 29801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:38.677654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.66:34943)
0109 21:08:38.678058 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:38.678130 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":86,"threads_started":1}
I20250109 21:08:38.886420 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 29309
I20250109 21:08:38.892616 29802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:38.892184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35275 (local address 127.12.126.67:41379)
0109 21:08:38.892451 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:38.892510 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.126:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
I20250109 21:08:38.894554 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 29484
I20250109 21:08:38.900450 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 29617
2025-01-09T21:08:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59386 ms)
[----------] 31 tests from SecurityITest (345507 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:08:39 dist-test-slave-g0zb krb5kdc[29805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:08:39 dist-test-slave-g0zb krb5kdc[29805](info): set up 2 sockets
Jan 09 21:08:39 dist-test-slave-g0zb krb5kdc[29805](info): commencing operation
krb5kdc: starting...
W20250109 21:08:40.051374 12793 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:08:40 dist-test-slave-g0zb krb5kdc[29805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:08:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:08:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:08:40.163620 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:35011
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:35011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:40.267655 29818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:40.267907 29818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:40.267985 29818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:40.271085 29818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:08:40.271144 29818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:40.271184 29818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:40.271227 29818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:08:40.271270 29818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:08:40.275414 29818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:35011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:35011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:40.276451 29818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:40.277272 29818 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:40.284022 29826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:40.284708 29824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:40.285519 29823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:40.286296 29818 server_base.cc:1034] running on GCE node
I20250109 21:08:40.286746 29818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:40.287658 29818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:40.288803 29818 hybrid_clock.cc:648] HybridClock initialized: now 1736456920288776 us; error 40 us; skew 500 ppm
Jan 09 21:08:40 dist-test-slave-g0zb krb5kdc[29805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456920, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:40.291440 29818 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:08:40.292306 29818 webserver.cc:458] Webserver started at http://127.12.126.126:42903/ using document root <none> and password file <none>
I20250109 21:08:40.292804 29818 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:40.292852 29818 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:40.293001 29818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:40.294677 29818 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "58c73948dd554f878b5529ba56b9cb8c"
format_stamp: "Formatted at 2025-01-09 21:08:40 on dist-test-slave-g0zb"
server_key: "6188a2c6161fe16305cb6002d0a4ea00"
server_key_iv: "d1de2c1e94cd55d574d43e75f4d1fbb1"
server_key_version: "encryptionkey@0"
I20250109 21:08:40.295171 29818 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "58c73948dd554f878b5529ba56b9cb8c"
format_stamp: "Formatted at 2025-01-09 21:08:40 on dist-test-slave-g0zb"
server_key: "6188a2c6161fe16305cb6002d0a4ea00"
server_key_iv: "d1de2c1e94cd55d574d43e75f4d1fbb1"
server_key_version: "encryptionkey@0"
I20250109 21:08:40.298296 29818 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250109 21:08:40.300359 29833 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:40.301306 29818 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:08:40.301420 29818 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "58c73948dd554f878b5529ba56b9cb8c"
format_stamp: "Formatted at 2025-01-09 21:08:40 on dist-test-slave-g0zb"
server_key: "6188a2c6161fe16305cb6002d0a4ea00"
server_key_iv: "d1de2c1e94cd55d574d43e75f4d1fbb1"
server_key_version: "encryptionkey@0"
I20250109 21:08:40.301502 29818 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:40.312783 29818 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:40.315312 29818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:40.315466 29818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:40.321635 29818 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:35011
I20250109 21:08:40.321870 29885 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:35011 every 8 connection(s)
I20250109 21:08:40.322592 29818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:08:40.324991 29886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:40.329010 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 29818
I20250109 21:08:40.329144 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:08:40.329375 12793 external_mini_cluster.cc:1453] Setting key 4ba288ec3c35cb492fe14a28fa8ec02a
I20250109 21:08:40.329857 29886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c: Bootstrap starting.
I20250109 21:08:40.332316 29886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:40.333137 29886 log.cc:826] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:40.334959 29886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c: No bootstrap required, opened a new log
Jan 09 21:08:40 dist-test-slave-g0zb krb5kdc[29805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:08:40.338061 29886 raft_consensus.cc:357] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c73948dd554f878b5529ba56b9cb8c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35011 } }
I20250109 21:08:40.338246 29886 raft_consensus.cc:383] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:40.338322 29886 raft_consensus.cc:738] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58c73948dd554f878b5529ba56b9cb8c, State: Initialized, Role: FOLLOWER
I20250109 21:08:40.338778 29886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c73948dd554f878b5529ba56b9cb8c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35011 } }
I20250109 21:08:40.338956 29886 raft_consensus.cc:397] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:08:40.339025 29886 raft_consensus.cc:491] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:08:40.339133 29886 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:40.340107 29886 raft_consensus.cc:513] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c73948dd554f878b5529ba56b9cb8c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35011 } }
I20250109 21:08:40.340440 29886 leader_election.cc:304] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58c73948dd554f878b5529ba56b9cb8c; no voters:
I20250109 21:08:40.340725 29886 leader_election.cc:290] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:08:40.340829 29891 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:40.341013 29891 raft_consensus.cc:695] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c [term 1 LEADER]: Becoming Leader. State: Replica: 58c73948dd554f878b5529ba56b9cb8c, State: Running, Role: LEADER
I20250109 21:08:40.341342 29891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c73948dd554f878b5529ba56b9cb8c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35011 } }
I20250109 21:08:40.341912 29886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:08:40.342342 29891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58c73948dd554f878b5529ba56b9cb8c. Latest consensus state: current_term: 1 leader_uuid: "58c73948dd554f878b5529ba56b9cb8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c73948dd554f878b5529ba56b9cb8c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35011 } } }
I20250109 21:08:40.342483 29891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:40.342733 29893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58c73948dd554f878b5529ba56b9cb8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c73948dd554f878b5529ba56b9cb8c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35011 } } }
I20250109 21:08:40.342836 29893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:40.342898 29895 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:08:40.343282 29889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:40.330585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.12.126.126:35011)
0109 21:08:40.331026 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:40.331041 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:40.331089 (+ 48us) server_negotiation.cc:407] Connection header received
0109 21:08:40.331786 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:40.331811 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:40.332172 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:40.332563 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:40.333414 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:40.334474 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:40.335171 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:40.335438 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:40.337491 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:40.337508 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:40.337518 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:40.337548 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:40.339723 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:40.340110 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:40.340117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:40.340125 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:40.340215 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:40.340439 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:40.340442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:40.340444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:40.340872 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:40.341108 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:40.341864 (+ 756us) server_negotiation.cc:299] Negotiation successful
0109 21:08:40.342129 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":97,"threads_started":1}
I20250109 21:08:40.345880 29895 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:08:40.351012 29895 catalog_manager.cc:1348] Generated new cluster ID: e6cf56aa8e1347a5bd105a619486407d
I20250109 21:08:40.351070 29895 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:08:40.359494 29895 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:08:40.360329 29895 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:08:40.375022 29895 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c: Generated new TSK 0
I20250109 21:08:40.375869 29895 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
I20250109 21:08:40.407377 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35011
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:40.511082 29912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:40.511284 29912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:40.511349 29912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:40.514611 29912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:40.514670 29912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:40.514744 29912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:08:40.518790 29912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:40.519759 29912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:40.520550 29912 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:40.529028 29920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:40.529695 29917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:40.529923 29918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:40.530710 29912 server_base.cc:1034] running on GCE node
I20250109 21:08:40.531057 29912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:40.531567 29912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:40.532709 29912 hybrid_clock.cc:648] HybridClock initialized: now 1736456920532689 us; error 41 us; skew 500 ppm
Jan 09 21:08:40 dist-test-slave-g0zb krb5kdc[29805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456920, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:40.535137 29912 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:08:40.535985 29912 webserver.cc:458] Webserver started at http://127.12.126.65:35027/ using document root <none> and password file <none>
I20250109 21:08:40.536602 29912 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:40.536679 29912 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:40.536880 29912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:40.538615 29912 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "7aac264551e14f31a490107afdb85f46"
format_stamp: "Formatted at 2025-01-09 21:08:40 on dist-test-slave-g0zb"
server_key: "6cdec212e95c3364232e4d95307b9636"
server_key_iv: "0faa6824e87742ed5ab01e35308411dd"
server_key_version: "encryptionkey@0"
I20250109 21:08:40.539124 29912 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "7aac264551e14f31a490107afdb85f46"
format_stamp: "Formatted at 2025-01-09 21:08:40 on dist-test-slave-g0zb"
server_key: "6cdec212e95c3364232e4d95307b9636"
server_key_iv: "0faa6824e87742ed5ab01e35308411dd"
server_key_version: "encryptionkey@0"
I20250109 21:08:40.542304 29912 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250109 21:08:40.544390 29927 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:40.545411 29912 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:08:40.545522 29912 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "7aac264551e14f31a490107afdb85f46"
format_stamp: "Formatted at 2025-01-09 21:08:40 on dist-test-slave-g0zb"
server_key: "6cdec212e95c3364232e4d95307b9636"
server_key_iv: "0faa6824e87742ed5ab01e35308411dd"
server_key_version: "encryptionkey@0"
I20250109 21:08:40.545621 29912 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:40.554566 29912 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:40.556923 29912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:40.557101 29912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:40.557559 29912 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:40.558291 29912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:40.558344 29912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:40.558398 29912 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:40.558422 29912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:40.566279 29912 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:42241
I20250109 21:08:40.566324 30040 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:42241 every 8 connection(s)
I20250109 21:08:40.567219 29912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:08:40.572608 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 29912
I20250109 21:08:40.572732 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:08:40.573006 12793 external_mini_cluster.cc:1453] Setting key 46f4e838c376194e090467bf1a51bc1c
Jan 09 21:08:40 dist-test-slave-g0zb krb5kdc[29805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456920, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:08:40.579249 29889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:40.568769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:45925 (local address 127.12.126.126:35011)
0109 21:08:40.568891 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:40.568895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:40.569530 (+ 635us) server_negotiation.cc:407] Connection header received
0109 21:08:40.570654 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:40.570657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:40.570707 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:40.570816 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:40.572031 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:40.572536 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:40.573372 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:40.573555 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:40.575717 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:40.575739 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:40.575746 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:40.575788 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:40.577480 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:40.577938 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:40.577942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:40.577943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:40.577987 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:40.578257 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:40.578260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:40.578262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:40.578394 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:40.578478 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:40.578944 (+ 466us) server_negotiation.cc:299] Negotiation successful
0109 21:08:40.579076 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250109 21:08:40.579823 30043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:40.568993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35011 (local address 127.12.126.65:45925)
0109 21:08:40.569375 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:40.569425 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:40.570449 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:40.570949 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:40.570955 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:40.571302 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:40.571806 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:40.571818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:40.572658 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:40.572663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:40.573240 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:40.573251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:40.573467 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:40.574055 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:08:40.574077 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:40.575572 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:40.577590 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:40.577595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:40.577603 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:40.577832 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:40.578071 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:40.578074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:40.578077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:40.578160 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:40.578491 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:40.578498 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:40.578701 (+ 203us) client_negotiation.cc:769] Sending connection context
0109 21:08:40.578902 (+ 201us) client_negotiation.cc:240] Negotiation successful
0109 21:08:40.579155 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":92,"threads_started":1}
I20250109 21:08:40.580930 30041 heartbeater.cc:346] Connected to a master server at 127.12.126.126:35011
I20250109 21:08:40.581171 30041 heartbeater.cc:463] Registering TS with master...
WARNING: no policy specified for kudu/127.12.126.66@KRBTEST.COM; defaulting to no policy
I20250109 21:08:40.581761 30041 heartbeater.cc:510] Master 127.12.126.126:35011 requested a full tablet report, sending...
I20250109 21:08:40.583220 29850 ts_manager.cc:194] Registered new tserver with Master: 7aac264551e14f31a490107afdb85f46 (127.12.126.65:42241)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.66@KRBTEST.COM" created.
I20250109 21:08:40.584429 29850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:45925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
I20250109 21:08:40.597895 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35011
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:40.697096 30046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:40.697296 30046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:40.697356 30046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:40.700309 30046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:40.700357 30046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:40.700462 30046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:08:40.704598 30046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:40.705518 30046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:40.706310 30046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:40.714191 30052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:40.714877 30054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:40.715397 30051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:40.716120 30046 server_base.cc:1034] running on GCE node
I20250109 21:08:40.716431 30046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:40.716918 30046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:40.718062 30046 hybrid_clock.cc:648] HybridClock initialized: now 1736456920718045 us; error 36 us; skew 500 ppm
Jan 09 21:08:40 dist-test-slave-g0zb krb5kdc[29805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456920, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:40.720616 30046 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:08:40.721446 30046 webserver.cc:458] Webserver started at http://127.12.126.66:42377/ using document root <none> and password file <none>
I20250109 21:08:40.721978 30046 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:40.722043 30046 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:40.722225 30046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:40.723946 30046 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "a83530301afc44e38924cb8c21aa8e9f"
format_stamp: "Formatted at 2025-01-09 21:08:40 on dist-test-slave-g0zb"
server_key: "b161c028dc3aba07ec721b1baa24caa7"
server_key_iv: "91709e26dbcbe3d21a4614b8cac434ec"
server_key_version: "encryptionkey@0"
I20250109 21:08:40.724416 30046 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "a83530301afc44e38924cb8c21aa8e9f"
format_stamp: "Formatted at 2025-01-09 21:08:40 on dist-test-slave-g0zb"
server_key: "b161c028dc3aba07ec721b1baa24caa7"
server_key_iv: "91709e26dbcbe3d21a4614b8cac434ec"
server_key_version: "encryptionkey@0"
I20250109 21:08:40.727525 30046 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:08:40.729645 30061 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:40.730563 30046 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:40.730669 30046 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "a83530301afc44e38924cb8c21aa8e9f"
format_stamp: "Formatted at 2025-01-09 21:08:40 on dist-test-slave-g0zb"
server_key: "b161c028dc3aba07ec721b1baa24caa7"
server_key_iv: "91709e26dbcbe3d21a4614b8cac434ec"
server_key_version: "encryptionkey@0"
I20250109 21:08:40.730754 30046 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:40.760102 30046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:40.762670 30046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:40.762883 30046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:40.763407 30046 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:40.764149 30046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:40.764196 30046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:40.764261 30046 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:40.764292 30046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:40.772300 30046 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:43553
I20250109 21:08:40.772331 30174 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:43553 every 8 connection(s)
I20250109 21:08:40.773170 30046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
Jan 09 21:08:40 dist-test-slave-g0zb krb5kdc[29805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456920, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:08:40.783350 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 30046
I20250109 21:08:40.783473 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:08:40.783659 29889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:40.774660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:35811 (local address 127.12.126.126:35011)
0109 21:08:40.774846 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:40.774851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:40.775465 (+ 614us) server_negotiation.cc:407] Connection header received
0109 21:08:40.776579 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:40.776582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:40.776631 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:40.776705 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:40.777804 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:40.778298 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:40.778898 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:40.779015 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:40.780864 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:40.780881 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:40.780883 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:40.780907 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:40.782106 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:40.782501 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:40.782504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:40.782505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:40.782538 (+ 33us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:40.782720 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:40.782721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:40.782722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:40.782882 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:40.782958 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:40.783408 (+ 450us) server_negotiation.cc:299] Negotiation successful
0109 21:08:40.783533 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20250109 21:08:40.783777 12793 external_mini_cluster.cc:1453] Setting key 9b4bea02f610902dc6583131800ee08d
I20250109 21:08:40.784322 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:40.774937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35011 (local address 127.12.126.66:35811)
0109 21:08:40.775342 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:40.775374 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:40.776394 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:40.776839 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:40.776846 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:40.777192 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:40.777698 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:40.777710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:40.778386 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:40.778388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:40.778747 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:40.778753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:40.778970 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:40.779417 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:08:40.779435 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:40.780722 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:40.782206 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:40.782210 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:40.782218 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:40.782429 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:40.782596 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:40.782598 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:40.782599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:40.782662 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:40.782961 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:40.782964 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:40.783182 (+ 218us) client_negotiation.cc:769] Sending connection context
0109 21:08:40.783394 (+ 212us) client_negotiation.cc:240] Negotiation successful
0109 21:08:40.783648 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":97,"threads_started":1}
I20250109 21:08:40.785326 30175 heartbeater.cc:346] Connected to a master server at 127.12.126.126:35011
I20250109 21:08:40.785526 30175 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:40.785975 30175 heartbeater.cc:510] Master 127.12.126.126:35011 requested a full tablet report, sending...
I20250109 21:08:40.786983 29850 ts_manager.cc:194] Registered new tserver with Master: a83530301afc44e38924cb8c21aa8e9f (127.12.126.66:43553)
I20250109 21:08:40.787520 29850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:35811
WARNING: no policy specified for kudu/127.12.126.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
I20250109 21:08:40.807813 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35011
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:40.908152 30180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:40.908381 30180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:40.908450 30180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:40.911681 30180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:40.911737 30180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:40.911839 30180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:08:40.915889 30180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:40.916838 30180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:40.917659 30180 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:40.925408 30188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:40.926398 30180 server_base.cc:1034] running on GCE node
W20250109 21:08:40.926983 30185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:40.927623 30186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:40.928087 30180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:40.928838 30180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:40.929983 30180 hybrid_clock.cc:648] HybridClock initialized: now 1736456920929967 us; error 29 us; skew 500 ppm
Jan 09 21:08:40 dist-test-slave-g0zb krb5kdc[29805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456920, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:40.933425 30180 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:08:40.934629 30180 webserver.cc:458] Webserver started at http://127.12.126.67:37541/ using document root <none> and password file <none>
I20250109 21:08:40.935268 30180 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:40.935331 30180 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:40.935503 30180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:40.937323 30180 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "4fe96f295830476390f05d5a7dcffc5e"
format_stamp: "Formatted at 2025-01-09 21:08:40 on dist-test-slave-g0zb"
server_key: "3117feb8bac907a1c7ea2094492b6ac5"
server_key_iv: "1688efb1045b4f6e7c5ab1dbd22a1611"
server_key_version: "encryptionkey@0"
I20250109 21:08:40.937825 30180 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "4fe96f295830476390f05d5a7dcffc5e"
format_stamp: "Formatted at 2025-01-09 21:08:40 on dist-test-slave-g0zb"
server_key: "3117feb8bac907a1c7ea2094492b6ac5"
server_key_iv: "1688efb1045b4f6e7c5ab1dbd22a1611"
server_key_version: "encryptionkey@0"
I20250109 21:08:40.941228 30180 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:08:40.943426 30195 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:40.944444 30180 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:08:40.944567 30180 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "4fe96f295830476390f05d5a7dcffc5e"
format_stamp: "Formatted at 2025-01-09 21:08:40 on dist-test-slave-g0zb"
server_key: "3117feb8bac907a1c7ea2094492b6ac5"
server_key_iv: "1688efb1045b4f6e7c5ab1dbd22a1611"
server_key_version: "encryptionkey@0"
I20250109 21:08:40.944653 30180 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:40.976819 30180 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:40.979481 30180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:40.979640 30180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:40.980191 30180 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:40.980978 30180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:40.981024 30180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:40.981087 30180 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:40.981114 30180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:40.989274 30180 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:46531
I20250109 21:08:40.989305 30308 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:46531 every 8 connection(s)
I20250109 21:08:40.990170 30180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:08:40.992533 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 30180
I20250109 21:08:40.992642 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:08:40.992882 12793 external_mini_cluster.cc:1453] Setting key 1b3dd49290e32d8bedc00abe630140ef
Jan 09 21:08:40 dist-test-slave-g0zb krb5kdc[29805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456920, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:08:41.001000 29889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:40.991712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:44693 (local address 127.12.126.126:35011)
0109 21:08:40.991834 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:40.991837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:40.992442 (+ 605us) server_negotiation.cc:407] Connection header received
0109 21:08:40.993801 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:40.993804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:40.993863 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:40.993942 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:40.995204 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:40.995677 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:40.996224 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:40.996340 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:40.998361 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:40.998374 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:40.998376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:40.998396 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:40.999606 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:41.000029 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:41.000031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:41.000033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:41.000066 (+ 33us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:41.000252 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:41.000254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:41.000255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:41.000375 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:41.000440 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:41.000804 (+ 364us) server_negotiation.cc:299] Negotiation successful
0109 21:08:41.000888 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250109 21:08:41.001576 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:40.991938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35011 (local address 127.12.126.67:44693)
0109 21:08:40.992310 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:40.992343 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:40.993590 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:40.994128 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:40.994136 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:40.994476 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:40.995086 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:40.995100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:40.995763 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:40.995765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:40.996147 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:40.996153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:40.996305 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:40.996835 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:08:40.996857 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:40.998227 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:40.999716 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:40.999720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:40.999728 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:40.999944 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:41.000123 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:41.000125 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:41.000127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:41.000194 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:41.000438 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:41.000442 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:41.000634 (+ 192us) client_negotiation.cc:769] Sending connection context
0109 21:08:41.000781 (+ 147us) client_negotiation.cc:240] Negotiation successful
0109 21:08:41.000970 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":93,"threads_started":1}
I20250109 21:08:41.002525 30309 heartbeater.cc:346] Connected to a master server at 127.12.126.126:35011
I20250109 21:08:41.002736 30309 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:41.003213 30309 heartbeater.cc:510] Master 127.12.126.126:35011 requested a full tablet report, sending...
I20250109 21:08:41.004315 29850 ts_manager.cc:194] Registered new tserver with Master: 4fe96f295830476390f05d5a7dcffc5e (127.12.126.67:46531)
I20250109 21:08:41.005044 29850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:44693
I20250109 21:08:41.006850 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 21:08:41.014072 29889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:41.008603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.12.126.126:35011)
0109 21:08:41.008713 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:41.008716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:41.008850 (+ 134us) server_negotiation.cc:407] Connection header received
0109 21:08:41.008947 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:41.008949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:41.008984 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:41.009049 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:41.009730 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.010204 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:41.010742 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.010899 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:41.011838 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:41.011850 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:41.011852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:41.011866 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:41.013019 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:41.013343 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:41.013346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:41.013347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:41.013378 (+ 31us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:41.013519 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:41.013521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:41.013522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:41.013654 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:41.013748 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:41.013890 (+ 142us) server_negotiation.cc:299] Negotiation successful
0109 21:08:41.013970 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250109 21:08:41.016414 29850 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:08:41.018822 29850 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:08:41.030781 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:41.026131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:43553 (local address 127.0.0.1:54446)
0109 21:08:41.027040 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:41.027088 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:41.027225 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:41.027930 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:41.027932 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:41.027944 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:41.028232 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:41.028241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.029534 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:41.029537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:41.030326 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:41.030337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.030422 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:41.030476 (+ 54us) client_negotiation.cc:769] Sending connection context
0109 21:08:41.030562 (+ 86us) client_negotiation.cc:240] Negotiation successful
0109 21:08:41.030640 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"spinlock_wait_cycles":4096,"thread_start_us":60,"threads_started":1}
I20250109 21:08:41.030992 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:41.026709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:42241 (local address 127.0.0.1:36470)
0109 21:08:41.027656 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:41.027668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:41.027747 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:41.028233 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:41.028236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:41.028253 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:41.028478 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:41.028488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.029752 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:41.029756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:41.030547 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:41.030558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.030654 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:41.030667 (+ 13us) client_negotiation.cc:769] Sending connection context
0109 21:08:41.030718 (+ 51us) client_negotiation.cc:240] Negotiation successful
0109 21:08:41.030767 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"thread_start_us":146,"threads_started":1}
I20250109 21:08:41.031126 30324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:41.026743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43392 (local address 127.12.126.67:46531)
0109 21:08:41.027176 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:41.027182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:41.027206 (+ 24us) server_negotiation.cc:407] Connection header received
0109 21:08:41.027411 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:41.027419 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:41.027582 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:41.027708 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:41.028393 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.029124 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:41.029956 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.030457 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:41.030561 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:41.030897 (+ 336us) server_negotiation.cc:299] Negotiation successful
0109 21:08:41.030992 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":126,"threads_started":1}
I20250109 21:08:41.031455 30321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:41.026251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54446 (local address 127.12.126.66:43553)
0109 21:08:41.027510 (+ 1259us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:41.027516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:41.027539 (+ 23us) server_negotiation.cc:407] Connection header received
0109 21:08:41.027630 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:41.027635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:41.027800 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:41.027951 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:41.028382 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.029415 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:41.030480 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.031055 (+ 575us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:41.031159 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:41.031230 (+ 71us) server_negotiation.cc:299] Negotiation successful
0109 21:08:41.031318 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1160,"thread_start_us":95,"threads_started":1}
I20250109 21:08:41.031700 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:41.026296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:46531 (local address 127.0.0.1:43392)
0109 21:08:41.027070 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:41.027098 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:41.027224 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:41.027875 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:41.027879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:41.027902 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:41.028232 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:41.028241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.029236 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:41.029240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:41.029817 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:41.029825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.030631 (+ 806us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:41.030647 (+ 16us) client_negotiation.cc:769] Sending connection context
0109 21:08:41.030709 (+ 62us) client_negotiation.cc:240] Negotiation successful
0109 21:08:41.030767 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"spinlock_wait_cycles":4864,"thread_start_us":71,"threads_started":1}
I20250109 21:08:41.032303 30325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:41.026923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36470 (local address 127.12.126.65:42241)
0109 21:08:41.027396 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:41.027402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:41.027860 (+ 458us) server_negotiation.cc:407] Connection header received
0109 21:08:41.027951 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:41.027956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:41.028104 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:41.028233 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:41.028606 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.029653 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:41.031408 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.031965 (+ 557us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:41.032045 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:41.032097 (+ 52us) server_negotiation.cc:299] Negotiation successful
0109 21:08:41.032176 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":105,"threads_started":1}
I20250109 21:08:41.033773 30109 tablet_service.cc:1467] Processing CreateTablet for tablet d79c0dcc7fbf42b28fb23670557b6f5a (DEFAULT_TABLE table=test-table [id=e6ce2f93bb9c4b9f80df5db60dbfd68d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:41.033947 30243 tablet_service.cc:1467] Processing CreateTablet for tablet d79c0dcc7fbf42b28fb23670557b6f5a (DEFAULT_TABLE table=test-table [id=e6ce2f93bb9c4b9f80df5db60dbfd68d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:41.034333 29975 tablet_service.cc:1467] Processing CreateTablet for tablet d79c0dcc7fbf42b28fb23670557b6f5a (DEFAULT_TABLE table=test-table [id=e6ce2f93bb9c4b9f80df5db60dbfd68d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:41.034641 30109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d79c0dcc7fbf42b28fb23670557b6f5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:41.034844 30243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d79c0dcc7fbf42b28fb23670557b6f5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:41.035157 29975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d79c0dcc7fbf42b28fb23670557b6f5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:41.039680 30326 tablet_bootstrap.cc:492] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f: Bootstrap starting.
I20250109 21:08:41.040722 30327 tablet_bootstrap.cc:492] T d79c0dcc7fbf42b28fb23670557b6f5a P 7aac264551e14f31a490107afdb85f46: Bootstrap starting.
I20250109 21:08:41.041560 30328 tablet_bootstrap.cc:492] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e: Bootstrap starting.
I20250109 21:08:41.042704 30326 tablet_bootstrap.cc:654] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:41.042950 30327 tablet_bootstrap.cc:654] T d79c0dcc7fbf42b28fb23670557b6f5a P 7aac264551e14f31a490107afdb85f46: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:41.043438 30328 tablet_bootstrap.cc:654] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:41.044169 30327 log.cc:826] T d79c0dcc7fbf42b28fb23670557b6f5a P 7aac264551e14f31a490107afdb85f46: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:41.044169 30326 log.cc:826] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:41.044450 30328 log.cc:826] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:41.045919 30326 tablet_bootstrap.cc:492] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f: No bootstrap required, opened a new log
I20250109 21:08:41.045919 30327 tablet_bootstrap.cc:492] T d79c0dcc7fbf42b28fb23670557b6f5a P 7aac264551e14f31a490107afdb85f46: No bootstrap required, opened a new log
I20250109 21:08:41.046098 30326 ts_tablet_manager.cc:1397] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250109 21:08:41.046098 30327 ts_tablet_manager.cc:1397] T d79c0dcc7fbf42b28fb23670557b6f5a P 7aac264551e14f31a490107afdb85f46: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20250109 21:08:41.047982 30328 tablet_bootstrap.cc:492] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e: No bootstrap required, opened a new log
I20250109 21:08:41.048100 30328 ts_tablet_manager.cc:1397] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250109 21:08:41.049340 30327 raft_consensus.cc:357] T d79c0dcc7fbf42b28fb23670557b6f5a P 7aac264551e14f31a490107afdb85f46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83530301afc44e38924cb8c21aa8e9f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43553 } } peers { permanent_uuid: "4fe96f295830476390f05d5a7dcffc5e" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 46531 } } peers { permanent_uuid: "7aac264551e14f31a490107afdb85f46" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42241 } }
I20250109 21:08:41.049340 30326 raft_consensus.cc:357] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83530301afc44e38924cb8c21aa8e9f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43553 } } peers { permanent_uuid: "4fe96f295830476390f05d5a7dcffc5e" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 46531 } } peers { permanent_uuid: "7aac264551e14f31a490107afdb85f46" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42241 } }
I20250109 21:08:41.049533 30327 raft_consensus.cc:383] T d79c0dcc7fbf42b28fb23670557b6f5a P 7aac264551e14f31a490107afdb85f46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:41.049533 30326 raft_consensus.cc:383] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:41.049583 30326 raft_consensus.cc:738] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a83530301afc44e38924cb8c21aa8e9f, State: Initialized, Role: FOLLOWER
I20250109 21:08:41.049583 30327 raft_consensus.cc:738] T d79c0dcc7fbf42b28fb23670557b6f5a P 7aac264551e14f31a490107afdb85f46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aac264551e14f31a490107afdb85f46, State: Initialized, Role: FOLLOWER
I20250109 21:08:41.050014 30327 consensus_queue.cc:260] T d79c0dcc7fbf42b28fb23670557b6f5a P 7aac264551e14f31a490107afdb85f46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83530301afc44e38924cb8c21aa8e9f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43553 } } peers { permanent_uuid: "4fe96f295830476390f05d5a7dcffc5e" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 46531 } } peers { permanent_uuid: "7aac264551e14f31a490107afdb85f46" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42241 } }
I20250109 21:08:41.050009 30326 consensus_queue.cc:260] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83530301afc44e38924cb8c21aa8e9f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43553 } } peers { permanent_uuid: "4fe96f295830476390f05d5a7dcffc5e" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 46531 } } peers { permanent_uuid: "7aac264551e14f31a490107afdb85f46" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42241 } }
I20250109 21:08:41.050638 30175 heartbeater.cc:502] Master 127.12.126.126:35011 was elected leader, sending a full tablet report...
I20250109 21:08:41.050839 30327 ts_tablet_manager.cc:1428] T d79c0dcc7fbf42b28fb23670557b6f5a P 7aac264551e14f31a490107afdb85f46: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20250109 21:08:41.050844 30326 ts_tablet_manager.cc:1428] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:08:41.050729 30328 raft_consensus.cc:357] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83530301afc44e38924cb8c21aa8e9f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43553 } } peers { permanent_uuid: "4fe96f295830476390f05d5a7dcffc5e" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 46531 } } peers { permanent_uuid: "7aac264551e14f31a490107afdb85f46" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42241 } }
I20250109 21:08:41.050921 30328 raft_consensus.cc:383] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:41.050976 30328 raft_consensus.cc:738] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe96f295830476390f05d5a7dcffc5e, State: Initialized, Role: FOLLOWER
I20250109 21:08:41.051151 30041 heartbeater.cc:502] Master 127.12.126.126:35011 was elected leader, sending a full tablet report...
I20250109 21:08:41.051343 30328 consensus_queue.cc:260] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83530301afc44e38924cb8c21aa8e9f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43553 } } peers { permanent_uuid: "4fe96f295830476390f05d5a7dcffc5e" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 46531 } } peers { permanent_uuid: "7aac264551e14f31a490107afdb85f46" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42241 } }
I20250109 21:08:41.052135 30328 ts_tablet_manager.cc:1428] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250109 21:08:41.052453 30309 heartbeater.cc:502] Master 127.12.126.126:35011 was elected leader, sending a full tablet report...
W20250109 21:08:41.068215 30042 tablet.cc:2367] T d79c0dcc7fbf42b28fb23670557b6f5a P 7aac264551e14f31a490107afdb85f46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250109 21:08:41.241036 30310 tablet.cc:2367] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:08:41.270462 30333 raft_consensus.cc:491] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:08:41.270670 30333 raft_consensus.cc:513] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83530301afc44e38924cb8c21aa8e9f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43553 } } peers { permanent_uuid: "4fe96f295830476390f05d5a7dcffc5e" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 46531 } } peers { permanent_uuid: "7aac264551e14f31a490107afdb85f46" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42241 } }
I20250109 21:08:41.271859 30333 leader_election.cc:290] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fe96f295830476390f05d5a7dcffc5e (127.12.126.67:46531), 7aac264551e14f31a490107afdb85f46 (127.12.126.65:42241)
W20250109 21:08:41.274124 30176 tablet.cc:2367] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:08:41.274667 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:41.272093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:46531 (local address 127.12.126.66:52681)
0109 21:08:41.272217 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:41.272233 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:41.272323 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:41.272618 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:41.272621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:41.272640 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:41.272890 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:41.272900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.273604 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:41.273608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:41.274280 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:41.274291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.274418 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:41.274438 (+ 20us) client_negotiation.cc:769] Sending connection context
0109 21:08:41.274489 (+ 51us) client_negotiation.cc:240] Negotiation successful
0109 21:08:41.274544 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20250109 21:08:41.275254 30324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:41.272155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:52681 (local address 127.12.126.67:46531)
0109 21:08:41.272331 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:41.272337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:41.272357 (+ 20us) server_negotiation.cc:407] Connection header received
0109 21:08:41.272412 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:41.272418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:41.272467 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:41.272536 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:41.273026 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.273498 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:41.274431 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.274960 (+ 529us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:41.275001 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:41.275055 (+ 54us) server_negotiation.cc:299] Negotiation successful
0109 21:08:41.275111 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20250109 21:08:41.275525 30335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:41.272095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:42241 (local address 127.12.126.66:39827)
0109 21:08:41.272468 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:41.272478 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:41.272567 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:41.272806 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:41.272809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:41.272822 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:41.273077 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:41.273084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.273758 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:41.273761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:41.274510 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:41.274521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.275268 (+ 747us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:41.275286 (+ 18us) client_negotiation.cc:769] Sending connection context
0109 21:08:41.275372 (+ 86us) client_negotiation.cc:240] Negotiation successful
0109 21:08:41.275433 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":112,"thread_start_us":108,"threads_started":1}
I20250109 21:08:41.275697 30325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:41.272213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:39827 (local address 127.12.126.65:42241)
0109 21:08:41.272333 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:41.272337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:41.272558 (+ 221us) server_negotiation.cc:407] Connection header received
0109 21:08:41.272676 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:41.272679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:41.272721 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:41.272781 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:41.273193 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.273648 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:41.274655 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.275142 (+ 487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:41.275195 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:41.275501 (+ 306us) server_negotiation.cc:299] Negotiation successful
0109 21:08:41.275581 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250109 21:08:41.275861 30263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d79c0dcc7fbf42b28fb23670557b6f5a" candidate_uuid: "a83530301afc44e38924cb8c21aa8e9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe96f295830476390f05d5a7dcffc5e" is_pre_election: true
I20250109 21:08:41.276166 30263 raft_consensus.cc:2463] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a83530301afc44e38924cb8c21aa8e9f in term 0.
I20250109 21:08:41.276249 29995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d79c0dcc7fbf42b28fb23670557b6f5a" candidate_uuid: "a83530301afc44e38924cb8c21aa8e9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aac264551e14f31a490107afdb85f46" is_pre_election: true
I20250109 21:08:41.276506 29995 raft_consensus.cc:2463] T d79c0dcc7fbf42b28fb23670557b6f5a P 7aac264551e14f31a490107afdb85f46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a83530301afc44e38924cb8c21aa8e9f in term 0.
I20250109 21:08:41.276607 30064 leader_election.cc:304] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe96f295830476390f05d5a7dcffc5e, a83530301afc44e38924cb8c21aa8e9f; no voters:
I20250109 21:08:41.276862 30333 raft_consensus.cc:2798] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:08:41.276943 30333 raft_consensus.cc:491] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:08:41.276983 30333 raft_consensus.cc:3054] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:41.277943 30333 raft_consensus.cc:513] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83530301afc44e38924cb8c21aa8e9f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43553 } } peers { permanent_uuid: "4fe96f295830476390f05d5a7dcffc5e" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 46531 } } peers { permanent_uuid: "7aac264551e14f31a490107afdb85f46" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42241 } }
I20250109 21:08:41.278227 30333 leader_election.cc:290] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [CANDIDATE]: Term 1 election: Requested vote from peers 4fe96f295830476390f05d5a7dcffc5e (127.12.126.67:46531), 7aac264551e14f31a490107afdb85f46 (127.12.126.65:42241)
I20250109 21:08:41.278524 30263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d79c0dcc7fbf42b28fb23670557b6f5a" candidate_uuid: "a83530301afc44e38924cb8c21aa8e9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe96f295830476390f05d5a7dcffc5e"
I20250109 21:08:41.278568 29995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d79c0dcc7fbf42b28fb23670557b6f5a" candidate_uuid: "a83530301afc44e38924cb8c21aa8e9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aac264551e14f31a490107afdb85f46"
I20250109 21:08:41.278643 30263 raft_consensus.cc:3054] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:41.278667 29995 raft_consensus.cc:3054] T d79c0dcc7fbf42b28fb23670557b6f5a P 7aac264551e14f31a490107afdb85f46 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:41.279604 29995 raft_consensus.cc:2463] T d79c0dcc7fbf42b28fb23670557b6f5a P 7aac264551e14f31a490107afdb85f46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a83530301afc44e38924cb8c21aa8e9f in term 1.
I20250109 21:08:41.279626 30263 raft_consensus.cc:2463] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a83530301afc44e38924cb8c21aa8e9f in term 1.
I20250109 21:08:41.279901 30065 leader_election.cc:304] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aac264551e14f31a490107afdb85f46, a83530301afc44e38924cb8c21aa8e9f; no voters:
I20250109 21:08:41.280074 30333 raft_consensus.cc:2798] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:41.280287 30333 raft_consensus.cc:695] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [term 1 LEADER]: Becoming Leader. State: Replica: a83530301afc44e38924cb8c21aa8e9f, State: Running, Role: LEADER
I20250109 21:08:41.280586 30333 consensus_queue.cc:237] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83530301afc44e38924cb8c21aa8e9f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43553 } } peers { permanent_uuid: "4fe96f295830476390f05d5a7dcffc5e" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 46531 } } peers { permanent_uuid: "7aac264551e14f31a490107afdb85f46" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42241 } }
I20250109 21:08:41.283459 29848 catalog_manager.cc:5526] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f reported cstate change: term changed from 0 to 1, leader changed from <none> to a83530301afc44e38924cb8c21aa8e9f (127.12.126.66). New cstate: current_term: 1 leader_uuid: "a83530301afc44e38924cb8c21aa8e9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83530301afc44e38924cb8c21aa8e9f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fe96f295830476390f05d5a7dcffc5e" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 46531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7aac264551e14f31a490107afdb85f46" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42241 } health_report { overall_health: UNKNOWN } } }
I20250109 21:08:41.323800 30321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:41.320490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54454 (local address 127.12.126.66:43553)
0109 21:08:41.320614 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:41.320620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:41.320645 (+ 25us) server_negotiation.cc:407] Connection header received
0109 21:08:41.320727 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:41.320729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:41.320778 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:41.320880 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:08:41.321194 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.321883 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:41.322617 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.322836 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:41.322949 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:08:41.323446 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:08:41.323556 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:41.323633 (+ 77us) server_negotiation.cc:299] Negotiation successful
0109 21:08:41.323681 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250109 21:08:41.329483 29995 raft_consensus.cc:1270] T d79c0dcc7fbf42b28fb23670557b6f5a P 7aac264551e14f31a490107afdb85f46 [term 1 FOLLOWER]: Refusing update from remote peer a83530301afc44e38924cb8c21aa8e9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:08:41.329516 30263 raft_consensus.cc:1270] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e [term 1 FOLLOWER]: Refusing update from remote peer a83530301afc44e38924cb8c21aa8e9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:08:41.330163 30333 consensus_queue.cc:1035] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aac264551e14f31a490107afdb85f46" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 42241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:08:41.330353 30336 consensus_queue.cc:1035] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe96f295830476390f05d5a7dcffc5e" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 46531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:08:41.339200 30340 mvcc.cc:204] Tried to move back new op lower bound from 7112527549758918656 to 7112527549567987712. Current Snapshot: MvccSnapshot[applied={T|T < 7112527549758918656}]
I20250109 21:08:41.344648 29849 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45902:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:08:41.344774 29849 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45902:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:08:41.346786 29849 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 58c73948dd554f878b5529ba56b9cb8c: Sending DeleteTablet for 3 replicas of tablet d79c0dcc7fbf42b28fb23670557b6f5a
I20250109 21:08:41.347453 30109 tablet_service.cc:1514] Processing DeleteTablet for tablet d79c0dcc7fbf42b28fb23670557b6f5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:41 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:54446
I20250109 21:08:41.347608 30243 tablet_service.cc:1514] Processing DeleteTablet for tablet d79c0dcc7fbf42b28fb23670557b6f5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:41 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:43392
I20250109 21:08:41.347914 29975 tablet_service.cc:1514] Processing DeleteTablet for tablet d79c0dcc7fbf42b28fb23670557b6f5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:41 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:36470
I20250109 21:08:41.348070 30349 tablet_replica.cc:331] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f: stopping tablet replica
I20250109 21:08:41.348306 30349 raft_consensus.cc:2238] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:08:41.348428 30351 tablet_replica.cc:331] T d79c0dcc7fbf42b28fb23670557b6f5a P 7aac264551e14f31a490107afdb85f46: stopping tablet replica
I20250109 21:08:41.348555 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 29912
I20250109 21:08:41.348727 30351 raft_consensus.cc:2238] T d79c0dcc7fbf42b28fb23670557b6f5a P 7aac264551e14f31a490107afdb85f46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:08:41.349279 30350 tablet_replica.cc:331] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e: stopping tablet replica
I20250109 21:08:41.349515 30350 raft_consensus.cc:2238] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:08:41.349773 30350 raft_consensus.cc:2267] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:41.350535 30349 raft_consensus.cc:2267] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:41.351172 30350 ts_tablet_manager.cc:1905] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:08:41.351792 30349 ts_tablet_manager.cc:1905] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:08:41.354115 30349 ts_tablet_manager.cc:1918] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:08:41.354121 30350 ts_tablet_manager.cc:1918] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:08:41.354189 30349 log.cc:1198] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/d79c0dcc7fbf42b28fb23670557b6f5a
I20250109 21:08:41.354198 30350 log.cc:1198] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/d79c0dcc7fbf42b28fb23670557b6f5a
I20250109 21:08:41.354491 30349 ts_tablet_manager.cc:1939] T d79c0dcc7fbf42b28fb23670557b6f5a P a83530301afc44e38924cb8c21aa8e9f: Deleting consensus metadata
I20250109 21:08:41.354503 30350 ts_tablet_manager.cc:1939] T d79c0dcc7fbf42b28fb23670557b6f5a P 4fe96f295830476390f05d5a7dcffc5e: Deleting consensus metadata
I20250109 21:08:41.355542 29835 catalog_manager.cc:4872] TS a83530301afc44e38924cb8c21aa8e9f (127.12.126.66:43553): tablet d79c0dcc7fbf42b28fb23670557b6f5a (table test-table [id=e6ce2f93bb9c4b9f80df5db60dbfd68d]) successfully deleted
I20250109 21:08:41.355856 29836 catalog_manager.cc:4872] TS 4fe96f295830476390f05d5a7dcffc5e (127.12.126.67:46531): tablet d79c0dcc7fbf42b28fb23670557b6f5a (table test-table [id=e6ce2f93bb9c4b9f80df5db60dbfd68d]) successfully deleted
W20250109 21:08:41.357213 29837 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.126.65:42241 (error 108)
I20250109 21:08:41.357291 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 30046
I20250109 21:08:41.358486 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:41.358246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:42241 (local address 127.0.0.1:36474)
0109 21:08:41.358364 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:41.358412 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.65:42241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20250109 21:08:41.358593 29837 catalog_manager.cc:4615] TS 7aac264551e14f31a490107afdb85f46 (127.12.126.65:42241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d79c0dcc7fbf42b28fb23670557b6f5a: Network error: Client connection negotiation failed: client connection to 127.12.126.65:42241: connect: Connection refused (error 111)
I20250109 21:08:41.363719 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 30180
I20250109 21:08:41.369335 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 29818
2025-01-09T21:08:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2346 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2025-01-09T21:08:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:08:41Z Disabled control of system clock
I20250109 21:08:41.398295 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:35785
--webserver_interface=127.12.126.126
--webserver_port=0
--builtin_ntp_servers=127.12.126.84:45233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:35785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20250109 21:08:41.497073 30356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:41.497298 30356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:41.497342 30356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:41.500294 30356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:08:41.500355 30356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:41.500368 30356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:41.500385 30356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:08:41.500401 30356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:08:41.503911 30356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:45233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:35785
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:35785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:41.504784 30356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:41.505764 30356 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:41.513032 30362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:41.513510 30364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:41.513778 30361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:41.514611 30356 server_base.cc:1034] running on GCE node
I20250109 21:08:41.515064 30356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:41.515839 30356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:41.516979 30356 hybrid_clock.cc:648] HybridClock initialized: now 1736456921516963 us; error 33 us; skew 500 ppm
I20250109 21:08:41.518733 30356 webserver.cc:458] Webserver started at http://127.12.126.126:38771/ using document root <none> and password file <none>
I20250109 21:08:41.519265 30356 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:41.519341 30356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:41.519497 30356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:41.521131 30356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "fbb77211d94244d18109150ebfc5cca1"
format_stamp: "Formatted at 2025-01-09 21:08:41 on dist-test-slave-g0zb"
server_key: "d0d8abcd421dc68431db2115f005ad68"
server_key_iv: "d2420015e9c8869eb296d2a3ba73ea12"
server_key_version: "encryptionkey@0"
I20250109 21:08:41.521559 30356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "fbb77211d94244d18109150ebfc5cca1"
format_stamp: "Formatted at 2025-01-09 21:08:41 on dist-test-slave-g0zb"
server_key: "d0d8abcd421dc68431db2115f005ad68"
server_key_iv: "d2420015e9c8869eb296d2a3ba73ea12"
server_key_version: "encryptionkey@0"
I20250109 21:08:41.524732 30356 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:08:41.526623 30370 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:41.527580 30356 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:41.527684 30356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "fbb77211d94244d18109150ebfc5cca1"
format_stamp: "Formatted at 2025-01-09 21:08:41 on dist-test-slave-g0zb"
server_key: "d0d8abcd421dc68431db2115f005ad68"
server_key_iv: "d2420015e9c8869eb296d2a3ba73ea12"
server_key_version: "encryptionkey@0"
I20250109 21:08:41.527761 30356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:41.554123 30356 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:41.554684 30356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:41.554864 30356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:41.561139 30356 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:35785
I20250109 21:08:41.561151 30422 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:35785 every 8 connection(s)
I20250109 21:08:41.562211 30356 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:08:41.562587 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 30356
I20250109 21:08:41.562723 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:08:41.563046 12793 external_mini_cluster.cc:1453] Setting key faf281e76837ecae1bf10b3fda2f8742
I20250109 21:08:41.565194 30423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:41.570286 30423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1: Bootstrap starting.
I20250109 21:08:41.570783 30426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:41.564228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.12.126.126:35785)
0109 21:08:41.564843 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:41.564851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:41.564883 (+ 32us) server_negotiation.cc:407] Connection header received
0109 21:08:41.565834 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:41.565852 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:41.566172 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:41.566488 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:41.567002 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.567741 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:41.568396 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.568667 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:41.569216 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:41.569289 (+ 73us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:41.569299 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:41.569643 (+ 344us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:41.569681 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:41.569694 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:41.569787 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:41.569925 (+ 138us) server_negotiation.cc:299] Negotiation successful
0109 21:08:41.570239 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":143,"threads_started":1}
I20250109 21:08:41.572307 30423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:41.573048 30423 log.cc:826] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:41.574591 30423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1: No bootstrap required, opened a new log
I20250109 21:08:41.577500 30423 raft_consensus.cc:357] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb77211d94244d18109150ebfc5cca1" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35785 } }
I20250109 21:08:41.577677 30423 raft_consensus.cc:383] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:41.577778 30423 raft_consensus.cc:738] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb77211d94244d18109150ebfc5cca1, State: Initialized, Role: FOLLOWER
I20250109 21:08:41.578182 30423 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb77211d94244d18109150ebfc5cca1" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35785 } }
I20250109 21:08:41.578306 30423 raft_consensus.cc:397] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:08:41.578372 30423 raft_consensus.cc:491] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:08:41.578507 30423 raft_consensus.cc:3054] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:41.579313 30423 raft_consensus.cc:513] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb77211d94244d18109150ebfc5cca1" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35785 } }
I20250109 21:08:41.579595 30423 leader_election.cc:304] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbb77211d94244d18109150ebfc5cca1; no voters:
I20250109 21:08:41.579864 30423 leader_election.cc:290] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:08:41.579959 30428 raft_consensus.cc:2798] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:41.580191 30428 raft_consensus.cc:695] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1 [term 1 LEADER]: Becoming Leader. State: Replica: fbb77211d94244d18109150ebfc5cca1, State: Running, Role: LEADER
I20250109 21:08:41.580513 30428 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb77211d94244d18109150ebfc5cca1" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35785 } }
I20250109 21:08:41.580889 30423 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:08:41.582094 30429 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbb77211d94244d18109150ebfc5cca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb77211d94244d18109150ebfc5cca1" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35785 } } }
I20250109 21:08:41.582093 30430 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbb77211d94244d18109150ebfc5cca1. Latest consensus state: current_term: 1 leader_uuid: "fbb77211d94244d18109150ebfc5cca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb77211d94244d18109150ebfc5cca1" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35785 } } }
I20250109 21:08:41.582242 30430 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:41.582242 30429 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:41.583102 30438 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:08:41.585619 30438 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:08:41.590621 30438 catalog_manager.cc:1348] Generated new cluster ID: 500feec388e94d8d978160c07f9992a2
I20250109 21:08:41.590679 30438 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:08:41.614311 30438 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:08:41.615185 30438 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:08:41.619827 30438 catalog_manager.cc:5899] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1: Generated new TSK 0
I20250109 21:08:41.620458 30438 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:08:41.624935 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35785
--builtin_ntp_servers=127.12.126.84:45233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20250109 21:08:41.725687 30447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:41.725895 30447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:41.725946 30447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:41.729027 30447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:41.729074 30447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:41.729159 30447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:08:41.732802 30447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:45233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:41.733753 30447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:41.734823 30447 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:41.742374 30453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:41.743028 30452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:41.743609 30455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:41.744400 30447 server_base.cc:1034] running on GCE node
I20250109 21:08:41.744722 30447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:41.745219 30447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:41.746366 30447 hybrid_clock.cc:648] HybridClock initialized: now 1736456921746353 us; error 31 us; skew 500 ppm
I20250109 21:08:41.748106 30447 webserver.cc:458] Webserver started at http://127.12.126.65:38389/ using document root <none> and password file <none>
I20250109 21:08:41.748610 30447 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:41.748665 30447 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:41.748852 30447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:41.750489 30447 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "839ac2fb4cce4b719ddf47b7bb1ed8d9"
format_stamp: "Formatted at 2025-01-09 21:08:41 on dist-test-slave-g0zb"
server_key: "71a32931be79ecca91d6231019f879fe"
server_key_iv: "bc871b812a6e41b8410f9457e836062e"
server_key_version: "encryptionkey@0"
I20250109 21:08:41.750986 30447 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "839ac2fb4cce4b719ddf47b7bb1ed8d9"
format_stamp: "Formatted at 2025-01-09 21:08:41 on dist-test-slave-g0zb"
server_key: "71a32931be79ecca91d6231019f879fe"
server_key_iv: "bc871b812a6e41b8410f9457e836062e"
server_key_version: "encryptionkey@0"
I20250109 21:08:41.754190 30447 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20250109 21:08:41.756163 30461 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:41.757084 30447 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:08:41.757184 30447 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "839ac2fb4cce4b719ddf47b7bb1ed8d9"
format_stamp: "Formatted at 2025-01-09 21:08:41 on dist-test-slave-g0zb"
server_key: "71a32931be79ecca91d6231019f879fe"
server_key_iv: "bc871b812a6e41b8410f9457e836062e"
server_key_version: "encryptionkey@0"
I20250109 21:08:41.757264 30447 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:41.773084 30447 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:41.773579 30447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:41.773753 30447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:41.774250 30447 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:41.775075 30447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:41.775122 30447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:41.775192 30447 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:41.775228 30447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:41.783295 30447 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:44649
I20250109 21:08:41.783324 30574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:44649 every 8 connection(s)
I20250109 21:08:41.784142 30447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:08:41.790201 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 30447
I20250109 21:08:41.790338 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:08:41.790630 12793 external_mini_cluster.cc:1453] Setting key 5b89031b9453c6e0bbfc093a33d253d4
I20250109 21:08:41.791640 30426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:41.785637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:55293 (local address 127.12.126.126:35785)
0109 21:08:41.785790 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:41.785795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:41.786702 (+ 907us) server_negotiation.cc:407] Connection header received
0109 21:08:41.787481 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:41.787483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:41.787531 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:41.787606 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:41.789030 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.789525 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:41.790152 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.790282 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:41.791182 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:41.791200 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:41.791203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:41.791216 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:41.791233 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:41.791240 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:41.791301 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:41.791448 (+ 147us) server_negotiation.cc:299] Negotiation successful
0109 21:08:41.791506 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250109 21:08:41.792230 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:41.785920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35785 (local address 127.12.126.65:55293)
0109 21:08:41.786582 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:41.786615 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:41.787311 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:41.787737 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:41.787744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:41.787820 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:41.788745 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:41.788758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.789616 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:41.789618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:41.790051 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:41.790062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.790296 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:41.790968 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:08:41.790969 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:41.790999 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:08:41.791049 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:41.791311 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:41.791321 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:41.791326 (+ 5us) client_negotiation.cc:769] Sending connection context
0109 21:08:41.791442 (+ 116us) client_negotiation.cc:240] Negotiation successful
0109 21:08:41.791641 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
I20250109 21:08:41.792591 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35785
--builtin_ntp_servers=127.12.126.84:45233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20250109 21:08:41.793299 30575 heartbeater.cc:346] Connected to a master server at 127.12.126.126:35785
I20250109 21:08:41.793503 30575 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:41.794118 30575 heartbeater.cc:510] Master 127.12.126.126:35785 requested a full tablet report, sending...
I20250109 21:08:41.795642 30387 ts_manager.cc:194] Registered new tserver with Master: 839ac2fb4cce4b719ddf47b7bb1ed8d9 (127.12.126.65:44649)
I20250109 21:08:41.797340 30387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.126.65:55293
W20250109 21:08:41.893651 30578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:41.893846 30578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:41.893891 30578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:41.896795 30578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:41.896842 30578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:41.896925 30578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:08:41.900539 30578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:45233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:41.901521 30578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:41.902604 30578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:41.910691 30583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:41.911315 30584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:41.911898 30586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:41.912637 30578 server_base.cc:1034] running on GCE node
I20250109 21:08:41.913012 30578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:41.913532 30578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:41.914680 30578 hybrid_clock.cc:648] HybridClock initialized: now 1736456921914667 us; error 34 us; skew 500 ppm
I20250109 21:08:41.916484 30578 webserver.cc:458] Webserver started at http://127.12.126.66:38073/ using document root <none> and password file <none>
I20250109 21:08:41.917064 30578 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:41.917140 30578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:41.917320 30578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:41.919080 30578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "d7ed68f2d6294161810e29b853e23a3f"
format_stamp: "Formatted at 2025-01-09 21:08:41 on dist-test-slave-g0zb"
server_key: "8e75b55747ab298ca8a922bde99a4e60"
server_key_iv: "58b30800c0f3edadc956ddf78e60d1f4"
server_key_version: "encryptionkey@0"
I20250109 21:08:41.919481 30578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "d7ed68f2d6294161810e29b853e23a3f"
format_stamp: "Formatted at 2025-01-09 21:08:41 on dist-test-slave-g0zb"
server_key: "8e75b55747ab298ca8a922bde99a4e60"
server_key_iv: "58b30800c0f3edadc956ddf78e60d1f4"
server_key_version: "encryptionkey@0"
I20250109 21:08:41.922603 30578 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250109 21:08:41.924657 30592 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:41.925607 30578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250109 21:08:41.925726 30578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "d7ed68f2d6294161810e29b853e23a3f"
format_stamp: "Formatted at 2025-01-09 21:08:41 on dist-test-slave-g0zb"
server_key: "8e75b55747ab298ca8a922bde99a4e60"
server_key_iv: "58b30800c0f3edadc956ddf78e60d1f4"
server_key_version: "encryptionkey@0"
I20250109 21:08:41.925838 30578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:41.955552 30578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:41.956197 30578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:41.956362 30578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:41.956892 30578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:41.957813 30578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:41.957862 30578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:41.957927 30578 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:41.957957 30578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:41.966181 30578 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:42621
I20250109 21:08:41.966216 30705 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:42621 every 8 connection(s)
I20250109 21:08:41.967150 30578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:08:41.967936 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 30578
I20250109 21:08:41.968053 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:08:41.968338 12793 external_mini_cluster.cc:1453] Setting key a45f9f7d6d8103a682830897c3b0644a
I20250109 21:08:41.970554 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35785
--builtin_ntp_servers=127.12.126.84:45233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20250109 21:08:41.975826 30426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:41.968949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:58479 (local address 127.12.126.126:35785)
0109 21:08:41.969110 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:41.969114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:41.970120 (+ 1006us) server_negotiation.cc:407] Connection header received
0109 21:08:41.971021 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:41.971024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:41.971088 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:41.971166 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:41.972870 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.973630 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:41.974322 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.974561 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:41.975245 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:41.975269 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:41.975275 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:41.975292 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:41.975319 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:41.975334 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:41.975420 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:41.975549 (+ 129us) server_negotiation.cc:299] Negotiation successful
0109 21:08:41.975622 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20250109 21:08:41.976441 30708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:41.969283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35785 (local address 127.12.126.66:58479)
0109 21:08:41.969994 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:41.970035 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:41.970778 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:41.971396 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:41.971409 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:41.971515 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:41.972724 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:41.972737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.973765 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:41.973767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:41.974197 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:41.974205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:41.974376 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:41.975019 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:08:41.975021 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:41.975053 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:08:41.975108 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:41.975424 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:41.975428 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:41.975432 (+ 4us) client_negotiation.cc:769] Sending connection context
0109 21:08:41.975524 (+ 92us) client_negotiation.cc:240] Negotiation successful
0109 21:08:41.975697 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
I20250109 21:08:41.977504 30706 heartbeater.cc:346] Connected to a master server at 127.12.126.126:35785
I20250109 21:08:41.977710 30706 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:41.978212 30706 heartbeater.cc:510] Master 127.12.126.126:35785 requested a full tablet report, sending...
I20250109 21:08:41.979207 30387 ts_manager.cc:194] Registered new tserver with Master: d7ed68f2d6294161810e29b853e23a3f (127.12.126.66:42621)
I20250109 21:08:41.979941 30387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.126.66:58479
W20250109 21:08:42.076952 30709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:42.077190 30709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:42.077240 30709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:42.080130 30709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:42.080178 30709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:42.080241 30709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:08:42.083866 30709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:45233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:35785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:42.084969 30709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:42.086208 30709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:42.094365 30714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:42.094457 30717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:42.094947 30715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:42.095654 30709 server_base.cc:1034] running on GCE node
I20250109 21:08:42.095995 30709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:42.096498 30709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:42.097636 30709 hybrid_clock.cc:648] HybridClock initialized: now 1736456922097620 us; error 38 us; skew 500 ppm
I20250109 21:08:42.099496 30709 webserver.cc:458] Webserver started at http://127.12.126.67:43647/ using document root <none> and password file <none>
I20250109 21:08:42.100050 30709 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:42.100128 30709 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:42.100294 30709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:42.102015 30709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "9952194efb8e46d5b8eeb4142a528513"
format_stamp: "Formatted at 2025-01-09 21:08:42 on dist-test-slave-g0zb"
server_key: "1679c988cbc531da1ec8fbe6734ed23b"
server_key_iv: "a9494ce61b4c9d6a4522cf659ef0933b"
server_key_version: "encryptionkey@0"
I20250109 21:08:42.102476 30709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "9952194efb8e46d5b8eeb4142a528513"
format_stamp: "Formatted at 2025-01-09 21:08:42 on dist-test-slave-g0zb"
server_key: "1679c988cbc531da1ec8fbe6734ed23b"
server_key_iv: "a9494ce61b4c9d6a4522cf659ef0933b"
server_key_version: "encryptionkey@0"
I20250109 21:08:42.105633 30709 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20250109 21:08:42.107880 30723 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:42.108894 30709 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:08:42.108986 30709 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "9952194efb8e46d5b8eeb4142a528513"
format_stamp: "Formatted at 2025-01-09 21:08:42 on dist-test-slave-g0zb"
server_key: "1679c988cbc531da1ec8fbe6734ed23b"
server_key_iv: "a9494ce61b4c9d6a4522cf659ef0933b"
server_key_version: "encryptionkey@0"
I20250109 21:08:42.109078 30709 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:42.122099 30709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:42.122651 30709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:42.122859 30709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:42.123368 30709 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:42.124228 30709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:42.124279 30709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:42.124336 30709 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:42.124369 30709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:42.132586 30709 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:45953
I20250109 21:08:42.132611 30836 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:45953 every 8 connection(s)
I20250109 21:08:42.133482 30709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:08:42.136487 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 30709
I20250109 21:08:42.136582 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:08:42.136853 12793 external_mini_cluster.cc:1453] Setting key 3c53e3a2e1ef1bf034e2d1cc5964f811
I20250109 21:08:42.140998 30426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:42.135077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:35579 (local address 127.12.126.126:35785)
0109 21:08:42.135212 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:42.135215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:42.136094 (+ 879us) server_negotiation.cc:407] Connection header received
0109 21:08:42.136859 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:42.136864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:42.136928 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:42.137015 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:42.138461 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.139064 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:42.139635 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.139821 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:42.140557 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:42.140575 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:42.140578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:42.140588 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:42.140606 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:42.140613 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:42.140676 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:42.140809 (+ 133us) server_negotiation.cc:299] Negotiation successful
0109 21:08:42.140866 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250109 21:08:42.141644 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:42.135399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:35785 (local address 127.12.126.67:35579)
0109 21:08:42.135963 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:42.135995 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:42.136641 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:42.137150 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:42.137180 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:42.137275 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:42.138332 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:42.138346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.139177 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:42.139180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:42.139535 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:42.139542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.139733 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:42.140351 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:08:42.140353 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:42.140382 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:08:42.140434 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:42.140684 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:42.140693 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:42.140699 (+ 6us) client_negotiation.cc:769] Sending connection context
0109 21:08:42.140809 (+ 110us) client_negotiation.cc:240] Negotiation successful
0109 21:08:42.141047 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":91,"threads_started":1}
I20250109 21:08:42.142567 30837 heartbeater.cc:346] Connected to a master server at 127.12.126.126:35785
I20250109 21:08:42.142742 30837 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:42.143239 30837 heartbeater.cc:510] Master 127.12.126.126:35785 requested a full tablet report, sending...
I20250109 21:08:42.144133 30387 ts_manager.cc:194] Registered new tserver with Master: 9952194efb8e46d5b8eeb4142a528513 (127.12.126.67:45953)
I20250109 21:08:42.144699 30387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.126.67:35579
I20250109 21:08:42.150314 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 21:08:42.154939 30426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:42.152103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53544 (local address 127.12.126.126:35785)
0109 21:08:42.152231 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:42.152235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:42.152351 (+ 116us) server_negotiation.cc:407] Connection header received
0109 21:08:42.152479 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:42.152482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:42.152521 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:42.152594 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:42.152914 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.153360 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:42.153898 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.154031 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:42.154517 (+ 486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:42.154531 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:42.154533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:42.154544 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:42.154559 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:42.154567 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:42.154623 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:42.154689 (+ 66us) server_negotiation.cc:299] Negotiation successful
0109 21:08:42.154729 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250109 21:08:42.157343 30387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:08:42.159914 30387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:08:42.171893 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:42.167654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:45953 (local address 127.0.0.1:43472)
0109 21:08:42.167987 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:42.168030 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:42.168310 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:42.168879 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:42.168884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:42.168909 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:42.169223 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:42.169236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.170474 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:42.170477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:42.171397 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:42.171409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.171511 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:42.171563 (+ 52us) client_negotiation.cc:769] Sending connection context
0109 21:08:42.171651 (+ 88us) client_negotiation.cc:240] Negotiation successful
0109 21:08:42.171735 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":74,"threads_started":1}
I20250109 21:08:42.172493 30852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:42.168646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:42621 (local address 127.0.0.1:57598)
0109 21:08:42.168999 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:42.169013 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:42.169104 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:42.169996 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:42.169999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:42.170017 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:42.170250 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:42.170259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.171278 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:42.171282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:42.172154 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:42.172164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.172256 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:42.172277 (+ 21us) client_negotiation.cc:769] Sending connection context
0109 21:08:42.172325 (+ 48us) client_negotiation.cc:240] Negotiation successful
0109 21:08:42.172377 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":79,"threads_started":1}
I20250109 21:08:42.172837 30850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:42.168336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:44649 (local address 127.0.0.1:55410)
0109 21:08:42.168575 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:42.168588 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:42.168662 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:42.169825 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:42.169829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:42.169848 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:42.170080 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:42.170086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.171491 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:42.171494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:42.172301 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:42.172311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.172385 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:42.172401 (+ 16us) client_negotiation.cc:769] Sending connection context
0109 21:08:42.172438 (+ 37us) client_negotiation.cc:240] Negotiation successful
0109 21:08:42.172490 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":93,"threads_started":1}
I20250109 21:08:42.173209 30849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:42.167777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.12.126.67:45953)
0109 21:08:42.168213 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:42.168221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:42.168250 (+ 29us) server_negotiation.cc:407] Connection header received
0109 21:08:42.168500 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:42.168506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:42.168709 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:42.168880 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:42.169373 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.170312 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:42.172222 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.172750 (+ 528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:42.172904 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:42.172974 (+ 70us) server_negotiation.cc:299] Negotiation successful
0109 21:08:42.173070 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":195,"threads_started":1}
I20250109 21:08:42.173525 30853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:42.169224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57598 (local address 127.12.126.66:42621)
0109 21:08:42.169567 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:42.169572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:42.169592 (+ 20us) server_negotiation.cc:407] Connection header received
0109 21:08:42.169671 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:42.169675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:42.169830 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:42.170001 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:42.170381 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.171168 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:42.172312 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.173153 (+ 841us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:42.173263 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:42.173337 (+ 74us) server_negotiation.cc:299] Negotiation successful
0109 21:08:42.173419 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":133,"threads_started":1}
I20250109 21:08:42.174188 30851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:42.168535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55410 (local address 127.12.126.65:44649)
0109 21:08:42.169367 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:42.169373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:42.169397 (+ 24us) server_negotiation.cc:407] Connection header received
0109 21:08:42.169492 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:42.169496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:42.169669 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:42.169816 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:42.170187 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.171353 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:42.172619 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.173107 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:42.173250 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:42.173955 (+ 705us) server_negotiation.cc:299] Negotiation successful
0109 21:08:42.174050 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":122,"threads_started":1}
I20250109 21:08:42.175850 30771 tablet_service.cc:1467] Processing CreateTablet for tablet 27ffa109416b4c399ad0ab6e24838d51 (DEFAULT_TABLE table=test-table [id=3522f086315048d3ad32363a3ccb090c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:42.175850 30640 tablet_service.cc:1467] Processing CreateTablet for tablet 27ffa109416b4c399ad0ab6e24838d51 (DEFAULT_TABLE table=test-table [id=3522f086315048d3ad32363a3ccb090c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:42.176855 30771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27ffa109416b4c399ad0ab6e24838d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:42.176855 30640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27ffa109416b4c399ad0ab6e24838d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:42.176785 30509 tablet_service.cc:1467] Processing CreateTablet for tablet 27ffa109416b4c399ad0ab6e24838d51 (DEFAULT_TABLE table=test-table [id=3522f086315048d3ad32363a3ccb090c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:42.177556 30509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27ffa109416b4c399ad0ab6e24838d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:42.182955 30854 tablet_bootstrap.cc:492] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9: Bootstrap starting.
I20250109 21:08:42.183920 30855 tablet_bootstrap.cc:492] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513: Bootstrap starting.
I20250109 21:08:42.183920 30856 tablet_bootstrap.cc:492] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f: Bootstrap starting.
I20250109 21:08:42.185501 30854 tablet_bootstrap.cc:654] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:42.186002 30856 tablet_bootstrap.cc:654] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:42.186002 30855 tablet_bootstrap.cc:654] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:42.186494 30854 log.cc:826] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:42.187075 30855 log.cc:826] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:42.187319 30856 log.cc:826] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:42.188267 30854 tablet_bootstrap.cc:492] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9: No bootstrap required, opened a new log
I20250109 21:08:42.188433 30854 ts_tablet_manager.cc:1397] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250109 21:08:42.188814 30855 tablet_bootstrap.cc:492] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513: No bootstrap required, opened a new log
I20250109 21:08:42.188865 30856 tablet_bootstrap.cc:492] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f: No bootstrap required, opened a new log
I20250109 21:08:42.188983 30856 ts_tablet_manager.cc:1397] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:08:42.188983 30855 ts_tablet_manager.cc:1397] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:08:42.191212 30854 raft_consensus.cc:357] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9952194efb8e46d5b8eeb4142a528513" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45953 } } peers { permanent_uuid: "839ac2fb4cce4b719ddf47b7bb1ed8d9" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 44649 } } peers { permanent_uuid: "d7ed68f2d6294161810e29b853e23a3f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42621 } }
I20250109 21:08:42.191390 30854 raft_consensus.cc:383] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:42.191444 30854 raft_consensus.cc:738] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 839ac2fb4cce4b719ddf47b7bb1ed8d9, State: Initialized, Role: FOLLOWER
I20250109 21:08:42.191839 30854 consensus_queue.cc:260] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9952194efb8e46d5b8eeb4142a528513" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45953 } } peers { permanent_uuid: "839ac2fb4cce4b719ddf47b7bb1ed8d9" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 44649 } } peers { permanent_uuid: "d7ed68f2d6294161810e29b853e23a3f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42621 } }
I20250109 21:08:42.192157 30855 raft_consensus.cc:357] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9952194efb8e46d5b8eeb4142a528513" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45953 } } peers { permanent_uuid: "839ac2fb4cce4b719ddf47b7bb1ed8d9" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 44649 } } peers { permanent_uuid: "d7ed68f2d6294161810e29b853e23a3f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42621 } }
I20250109 21:08:42.192157 30856 raft_consensus.cc:357] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9952194efb8e46d5b8eeb4142a528513" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45953 } } peers { permanent_uuid: "839ac2fb4cce4b719ddf47b7bb1ed8d9" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 44649 } } peers { permanent_uuid: "d7ed68f2d6294161810e29b853e23a3f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42621 } }
I20250109 21:08:42.192348 30855 raft_consensus.cc:383] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:42.192348 30856 raft_consensus.cc:383] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:42.192406 30855 raft_consensus.cc:738] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9952194efb8e46d5b8eeb4142a528513, State: Initialized, Role: FOLLOWER
I20250109 21:08:42.192406 30856 raft_consensus.cc:738] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ed68f2d6294161810e29b853e23a3f, State: Initialized, Role: FOLLOWER
I20250109 21:08:42.192596 30575 heartbeater.cc:502] Master 127.12.126.126:35785 was elected leader, sending a full tablet report...
I20250109 21:08:42.192696 30854 ts_tablet_manager.cc:1428] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250109 21:08:42.192790 30856 consensus_queue.cc:260] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9952194efb8e46d5b8eeb4142a528513" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45953 } } peers { permanent_uuid: "839ac2fb4cce4b719ddf47b7bb1ed8d9" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 44649 } } peers { permanent_uuid: "d7ed68f2d6294161810e29b853e23a3f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42621 } }
I20250109 21:08:42.192837 30855 consensus_queue.cc:260] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9952194efb8e46d5b8eeb4142a528513" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45953 } } peers { permanent_uuid: "839ac2fb4cce4b719ddf47b7bb1ed8d9" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 44649 } } peers { permanent_uuid: "d7ed68f2d6294161810e29b853e23a3f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42621 } }
I20250109 21:08:42.193620 30856 ts_tablet_manager.cc:1428] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:08:42.193620 30855 ts_tablet_manager.cc:1428] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20250109 21:08:42.193630 30837 heartbeater.cc:502] Master 127.12.126.126:35785 was elected leader, sending a full tablet report...
I20250109 21:08:42.194087 30706 heartbeater.cc:502] Master 127.12.126.126:35785 was elected leader, sending a full tablet report...
W20250109 21:08:42.219058 30707 tablet.cc:2367] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:08:42.258018 30860 raft_consensus.cc:491] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:08:42.258227 30860 raft_consensus.cc:513] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9952194efb8e46d5b8eeb4142a528513" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45953 } } peers { permanent_uuid: "839ac2fb4cce4b719ddf47b7bb1ed8d9" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 44649 } } peers { permanent_uuid: "d7ed68f2d6294161810e29b853e23a3f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42621 } }
I20250109 21:08:42.259363 30860 leader_election.cc:290] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9952194efb8e46d5b8eeb4142a528513 (127.12.126.67:45953), d7ed68f2d6294161810e29b853e23a3f (127.12.126.66:42621)
I20250109 21:08:42.262508 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:42.259612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:45953 (local address 127.12.126.65:60477)
0109 21:08:42.259730 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:42.259748 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:42.259852 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:42.260149 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:42.260152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:42.260183 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:42.260437 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:42.260447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.261339 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:42.261342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:42.262129 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:42.262139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.262251 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:42.262272 (+ 21us) client_negotiation.cc:769] Sending connection context
0109 21:08:42.262319 (+ 47us) client_negotiation.cc:240] Negotiation successful
0109 21:08:42.262373 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20250109 21:08:42.262976 30849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:42.259702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:60477 (local address 127.12.126.67:45953)
0109 21:08:42.259853 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:42.259858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:42.259872 (+ 14us) server_negotiation.cc:407] Connection header received
0109 21:08:42.259947 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:42.259950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:42.260002 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:42.260075 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:42.260549 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.261242 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:42.262256 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.262675 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:42.262715 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:42.262776 (+ 61us) server_negotiation.cc:299] Negotiation successful
0109 21:08:42.262863 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20250109 21:08:42.263423 30863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:42.259898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:42621 (local address 127.12.126.65:60763)
0109 21:08:42.260151 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:42.260164 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:42.260238 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:42.260459 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:42.260461 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:42.260483 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:42.260696 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:42.260702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.261613 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:42.261616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:42.262438 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:42.262446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.263172 (+ 726us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:42.263191 (+ 19us) client_negotiation.cc:769] Sending connection context
0109 21:08:42.263274 (+ 83us) client_negotiation.cc:240] Negotiation successful
0109 21:08:42.263325 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":87,"threads_started":1}
I20250109 21:08:42.263620 30853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:42.260052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:60763 (local address 127.12.126.66:42621)
0109 21:08:42.260194 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:42.260197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:42.260208 (+ 11us) server_negotiation.cc:407] Connection header received
0109 21:08:42.260334 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:42.260337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:42.260377 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:42.260440 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:42.260803 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.261498 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:42.262571 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.263064 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:42.263111 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:42.263319 (+ 208us) server_negotiation.cc:299] Negotiation successful
0109 21:08:42.263388 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250109 21:08:42.263561 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ffa109416b4c399ad0ab6e24838d51" candidate_uuid: "839ac2fb4cce4b719ddf47b7bb1ed8d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9952194efb8e46d5b8eeb4142a528513" is_pre_election: true
I20250109 21:08:42.263852 30791 raft_consensus.cc:2463] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 839ac2fb4cce4b719ddf47b7bb1ed8d9 in term 0.
I20250109 21:08:42.264159 30660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ffa109416b4c399ad0ab6e24838d51" candidate_uuid: "839ac2fb4cce4b719ddf47b7bb1ed8d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ed68f2d6294161810e29b853e23a3f" is_pre_election: true
I20250109 21:08:42.264302 30463 leader_election.cc:304] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 839ac2fb4cce4b719ddf47b7bb1ed8d9, 9952194efb8e46d5b8eeb4142a528513; no voters:
I20250109 21:08:42.264418 30660 raft_consensus.cc:2463] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 839ac2fb4cce4b719ddf47b7bb1ed8d9 in term 0.
I20250109 21:08:42.264585 30860 raft_consensus.cc:2798] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:08:42.264659 30860 raft_consensus.cc:491] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:08:42.264921 30860 raft_consensus.cc:3054] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:42.265861 30860 raft_consensus.cc:513] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9952194efb8e46d5b8eeb4142a528513" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45953 } } peers { permanent_uuid: "839ac2fb4cce4b719ddf47b7bb1ed8d9" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 44649 } } peers { permanent_uuid: "d7ed68f2d6294161810e29b853e23a3f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42621 } }
I20250109 21:08:42.266175 30860 leader_election.cc:290] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [CANDIDATE]: Term 1 election: Requested vote from peers 9952194efb8e46d5b8eeb4142a528513 (127.12.126.67:45953), d7ed68f2d6294161810e29b853e23a3f (127.12.126.66:42621)
I20250109 21:08:42.266500 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ffa109416b4c399ad0ab6e24838d51" candidate_uuid: "839ac2fb4cce4b719ddf47b7bb1ed8d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9952194efb8e46d5b8eeb4142a528513"
I20250109 21:08:42.266564 30660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ffa109416b4c399ad0ab6e24838d51" candidate_uuid: "839ac2fb4cce4b719ddf47b7bb1ed8d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ed68f2d6294161810e29b853e23a3f"
I20250109 21:08:42.266599 30791 raft_consensus.cc:3054] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:42.266702 30660 raft_consensus.cc:3054] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:42.267556 30791 raft_consensus.cc:2463] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 839ac2fb4cce4b719ddf47b7bb1ed8d9 in term 1.
I20250109 21:08:42.267701 30660 raft_consensus.cc:2463] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 839ac2fb4cce4b719ddf47b7bb1ed8d9 in term 1.
I20250109 21:08:42.267966 30463 leader_election.cc:304] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 839ac2fb4cce4b719ddf47b7bb1ed8d9, 9952194efb8e46d5b8eeb4142a528513; no voters:
I20250109 21:08:42.268144 30860 raft_consensus.cc:2798] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:42.268352 30860 raft_consensus.cc:695] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [term 1 LEADER]: Becoming Leader. State: Replica: 839ac2fb4cce4b719ddf47b7bb1ed8d9, State: Running, Role: LEADER
I20250109 21:08:42.268636 30860 consensus_queue.cc:237] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9952194efb8e46d5b8eeb4142a528513" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45953 } } peers { permanent_uuid: "839ac2fb4cce4b719ddf47b7bb1ed8d9" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 44649 } } peers { permanent_uuid: "d7ed68f2d6294161810e29b853e23a3f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42621 } }
I20250109 21:08:42.271464 30385 catalog_manager.cc:5526] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 839ac2fb4cce4b719ddf47b7bb1ed8d9 (127.12.126.65). New cstate: current_term: 1 leader_uuid: "839ac2fb4cce4b719ddf47b7bb1ed8d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9952194efb8e46d5b8eeb4142a528513" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "839ac2fb4cce4b719ddf47b7bb1ed8d9" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 44649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7ed68f2d6294161810e29b853e23a3f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42621 } health_report { overall_health: UNKNOWN } } }
W20250109 21:08:42.285315 30576 tablet.cc:2367] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:08:42.293763 30851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:42.290698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.12.126.65:44649)
0109 21:08:42.290868 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:42.290872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:42.290888 (+ 16us) server_negotiation.cc:407] Connection header received
0109 21:08:42.290986 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:42.290989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:42.291047 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:42.291154 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:08:42.291500 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.291993 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:42.292656 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:42.292861 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:42.292948 (+ 87us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:08:42.293355 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:08:42.293469 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:42.293573 (+ 104us) server_negotiation.cc:299] Negotiation successful
0109 21:08:42.293624 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250109 21:08:42.299146 30660 raft_consensus.cc:1270] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f [term 1 FOLLOWER]: Refusing update from remote peer 839ac2fb4cce4b719ddf47b7bb1ed8d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:08:42.299146 30791 raft_consensus.cc:1270] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513 [term 1 FOLLOWER]: Refusing update from remote peer 839ac2fb4cce4b719ddf47b7bb1ed8d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:08:42.299857 30864 consensus_queue.cc:1035] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9952194efb8e46d5b8eeb4142a528513" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 45953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:08:42.300036 30860 consensus_queue.cc:1035] T 27ffa109416b4c399ad0ab6e24838d51 P 839ac2fb4cce4b719ddf47b7bb1ed8d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7ed68f2d6294161810e29b853e23a3f" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:08:42.308661 30869 mvcc.cc:204] Tried to move back new op lower bound from 7112527553730301952 to 7112527553614528512. Current Snapshot: MvccSnapshot[applied={T|T < 7112527553730301952}]
I20250109 21:08:42.308928 30868 mvcc.cc:204] Tried to move back new op lower bound from 7112527553730301952 to 7112527553614528512. Current Snapshot: MvccSnapshot[applied={T|T < 7112527553730301952}]
I20250109 21:08:42.314435 30384 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53544:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:08:42.314586 30384 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53544:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:08:42.316884 30384 catalog_manager.cc:5813] T 00000000000000000000000000000000 P fbb77211d94244d18109150ebfc5cca1: Sending DeleteTablet for 3 replicas of tablet 27ffa109416b4c399ad0ab6e24838d51
I20250109 21:08:42.317554 30771 tablet_service.cc:1514] Processing DeleteTablet for tablet 27ffa109416b4c399ad0ab6e24838d51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:42 UTC) from {username='slave'} at 127.0.0.1:43472
I20250109 21:08:42.317629 30509 tablet_service.cc:1514] Processing DeleteTablet for tablet 27ffa109416b4c399ad0ab6e24838d51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:42 UTC) from {username='slave'} at 127.0.0.1:55410
I20250109 21:08:42.317760 30640 tablet_service.cc:1514] Processing DeleteTablet for tablet 27ffa109416b4c399ad0ab6e24838d51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:42 UTC) from {username='slave'} at 127.0.0.1:57598
I20250109 21:08:42.318056 30878 tablet_replica.cc:331] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513: stopping tablet replica
I20250109 21:08:42.318183 30879 tablet_replica.cc:331] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f: stopping tablet replica
I20250109 21:08:42.318394 30878 raft_consensus.cc:2238] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:08:42.318394 30879 raft_consensus.cc:2238] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:08:42.318688 30879 raft_consensus.cc:2267] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:42.318688 30878 raft_consensus.cc:2267] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:42.318779 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 30447
I20250109 21:08:42.320430 30878 ts_tablet_manager.cc:1905] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:08:42.321116 30879 ts_tablet_manager.cc:1905] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:08:42.323230 30878 ts_tablet_manager.cc:1918] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:08:42.323323 30878 log.cc:1198] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/27ffa109416b4c399ad0ab6e24838d51
I20250109 21:08:42.323559 30879 ts_tablet_manager.cc:1918] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:08:42.323654 30879 log.cc:1198] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/27ffa109416b4c399ad0ab6e24838d51
I20250109 21:08:42.323678 30878 ts_tablet_manager.cc:1939] T 27ffa109416b4c399ad0ab6e24838d51 P 9952194efb8e46d5b8eeb4142a528513: Deleting consensus metadata
I20250109 21:08:42.323967 30879 ts_tablet_manager.cc:1939] T 27ffa109416b4c399ad0ab6e24838d51 P d7ed68f2d6294161810e29b853e23a3f: Deleting consensus metadata
I20250109 21:08:42.324846 30372 catalog_manager.cc:4872] TS 9952194efb8e46d5b8eeb4142a528513 (127.12.126.67:45953): tablet 27ffa109416b4c399ad0ab6e24838d51 (table test-table [id=3522f086315048d3ad32363a3ccb090c]) successfully deleted
I20250109 21:08:42.325018 30372 catalog_manager.cc:4872] TS d7ed68f2d6294161810e29b853e23a3f (127.12.126.66:42621): tablet 27ffa109416b4c399ad0ab6e24838d51 (table test-table [id=3522f086315048d3ad32363a3ccb090c]) successfully deleted
W20250109 21:08:42.325877 30372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.126.65:44649 (error 108)
I20250109 21:08:42.326583 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 30578
I20250109 21:08:42.326865 30850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:42.326533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:44649 (local address 127.0.0.1:55440)
0109 21:08:42.326677 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:42.326746 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.65:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20250109 21:08:42.330905 30372 catalog_manager.cc:4615] TS 839ac2fb4cce4b719ddf47b7bb1ed8d9 (127.12.126.65:44649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27ffa109416b4c399ad0ab6e24838d51: Network error: Client connection negotiation failed: client connection to 127.12.126.65:44649: connect: Connection refused (error 111)
I20250109 21:08:42.332676 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 30709
I20250109 21:08:42.338433 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 30356
2025-01-09T21:08:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (968 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:08:42 dist-test-slave-g0zb krb5kdc[30883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:08:42 dist-test-slave-g0zb krb5kdc[30883](info): set up 2 sockets
Jan 09 21:08:42 dist-test-slave-g0zb krb5kdc[30883](info): commencing operation
krb5kdc: starting...
W20250109 21:08:43.363327 12793 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:08:43 dist-test-slave-g0zb krb5kdc[30883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:08:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:08:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:08:43.475514 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:37259
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:37259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:43.576678 30896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:43.576898 30896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:43.576946 30896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:43.580049 30896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:08:43.580099 30896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:43.580121 30896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:43.580140 30896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:08:43.580157 30896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:08:43.584184 30896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:37259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:37259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:43.585064 30896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:43.585813 30896 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:43.593151 30902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:43.593395 30901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:43.593978 30904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:43.594722 30896 server_base.cc:1034] running on GCE node
I20250109 21:08:43.595207 30896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:43.596074 30896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:43.597249 30896 hybrid_clock.cc:648] HybridClock initialized: now 1736456923597233 us; error 24 us; skew 500 ppm
Jan 09 21:08:43 dist-test-slave-g0zb krb5kdc[30883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456923, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:43.599889 30896 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:08:43.600719 30896 webserver.cc:458] Webserver started at http://127.12.126.126:46349/ using document root <none> and password file <none>
I20250109 21:08:43.601243 30896 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:43.601294 30896 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:43.601472 30896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:43.603295 30896 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "7946cf4ac44149088877f35dab850653"
format_stamp: "Formatted at 2025-01-09 21:08:43 on dist-test-slave-g0zb"
server_key: "eb7dfdc32684dc2a936d41be1680607a"
server_key_iv: "cc4a64428c179a63d5accfc6f797d749"
server_key_version: "encryptionkey@0"
I20250109 21:08:43.603757 30896 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "7946cf4ac44149088877f35dab850653"
format_stamp: "Formatted at 2025-01-09 21:08:43 on dist-test-slave-g0zb"
server_key: "eb7dfdc32684dc2a936d41be1680607a"
server_key_iv: "cc4a64428c179a63d5accfc6f797d749"
server_key_version: "encryptionkey@0"
I20250109 21:08:43.607117 30896 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:08:43.609232 30911 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:43.610339 30896 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:43.610443 30896 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "7946cf4ac44149088877f35dab850653"
format_stamp: "Formatted at 2025-01-09 21:08:43 on dist-test-slave-g0zb"
server_key: "eb7dfdc32684dc2a936d41be1680607a"
server_key_iv: "cc4a64428c179a63d5accfc6f797d749"
server_key_version: "encryptionkey@0"
I20250109 21:08:43.610533 30896 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:43.624257 30896 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:43.627111 30896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:43.627262 30896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:43.633180 30896 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:37259
I20250109 21:08:43.633412 30963 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:37259 every 8 connection(s)
I20250109 21:08:43.634104 30896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:08:43.636417 30964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:43.640123 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 30896
I20250109 21:08:43.640234 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:08:43.640472 12793 external_mini_cluster.cc:1453] Setting key c157d7e90caef600b9476b943caa4a50
I20250109 21:08:43.641279 30964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653: Bootstrap starting.
I20250109 21:08:43.643683 30964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:43.644382 30964 log.cc:826] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:43.646124 30964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653: No bootstrap required, opened a new log
Jan 09 21:08:43 dist-test-slave-g0zb krb5kdc[30883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:08:43.649320 30964 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7946cf4ac44149088877f35dab850653" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 37259 } }
I20250109 21:08:43.649538 30964 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:43.649607 30964 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7946cf4ac44149088877f35dab850653, State: Initialized, Role: FOLLOWER
I20250109 21:08:43.650074 30964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7946cf4ac44149088877f35dab850653" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 37259 } }
I20250109 21:08:43.650221 30964 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:08:43.650287 30964 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:08:43.650400 30964 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:43.651371 30964 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7946cf4ac44149088877f35dab850653" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 37259 } }
I20250109 21:08:43.651705 30964 leader_election.cc:304] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7946cf4ac44149088877f35dab850653; no voters:
I20250109 21:08:43.652026 30964 leader_election.cc:290] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:08:43.652166 30969 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:43.652365 30969 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653 [term 1 LEADER]: Becoming Leader. State: Replica: 7946cf4ac44149088877f35dab850653, State: Running, Role: LEADER
I20250109 21:08:43.652658 30969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7946cf4ac44149088877f35dab850653" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 37259 } }
I20250109 21:08:43.653143 30964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:08:43.653208 30967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:43.641598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.12.126.126:37259)
0109 21:08:43.642031 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:43.642042 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:43.642079 (+ 37us) server_negotiation.cc:407] Connection header received
0109 21:08:43.642797 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:43.642856 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:43.643179 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:43.643516 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:43.644499 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:43.645569 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:43.646207 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:43.646465 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:43.648427 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:43.648446 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:43.648459 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:43.648495 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:43.650484 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:43.650916 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:43.650922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:43.650927 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:43.651015 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:43.651226 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:43.651232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:43.651236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:43.651656 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:43.651875 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:43.652120 (+ 245us) server_negotiation.cc:299] Negotiation successful
0109 21:08:43.652503 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":134,"threads_started":1}
I20250109 21:08:43.654678 30970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7946cf4ac44149088877f35dab850653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7946cf4ac44149088877f35dab850653" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 37259 } } }
I20250109 21:08:43.654778 30970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:43.655102 30971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7946cf4ac44149088877f35dab850653. Latest consensus state: current_term: 1 leader_uuid: "7946cf4ac44149088877f35dab850653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7946cf4ac44149088877f35dab850653" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 37259 } } }
I20250109 21:08:43.655599 30971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653 [sys.catalog]: This master's current role is: LEADER
W20250109 21:08:43.657351 30984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:08:43.657409 30984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 21:08:43.657526 30983 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:08:43.658665 30983 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:08:43.663518 30983 catalog_manager.cc:1348] Generated new cluster ID: 3049a04943f94fcf8c7bea71cd1d7cde
I20250109 21:08:43.663583 30983 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:08:43.681532 30983 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:08:43.682381 30983 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:08:43.690001 30983 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653: Generated new TSK 0
I20250109 21:08:43.690639 30983 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
I20250109 21:08:43.718116 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:37259
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:43.818421 30990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:43.818631 30990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:43.818689 30990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:43.821498 30990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:43.821547 30990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:43.821631 30990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:08:43.825830 30990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:37259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:43.826772 30990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:43.827554 30990 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:43.834780 30995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:43.835388 30998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:43.835944 30996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:43.836654 30990 server_base.cc:1034] running on GCE node
I20250109 21:08:43.836969 30990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:43.837445 30990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:43.838587 30990 hybrid_clock.cc:648] HybridClock initialized: now 1736456923838576 us; error 26 us; skew 500 ppm
Jan 09 21:08:43 dist-test-slave-g0zb krb5kdc[30883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456923, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:43.847061 30990 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:08:43.847877 30990 webserver.cc:458] Webserver started at http://127.12.126.65:33207/ using document root <none> and password file <none>
I20250109 21:08:43.848373 30990 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:43.848433 30990 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:43.848592 30990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:43.850234 30990 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "dcfb1f7225bc449db7390c0d23854b1c"
format_stamp: "Formatted at 2025-01-09 21:08:43 on dist-test-slave-g0zb"
server_key: "57da9ce5f71e0eff88092cd304be3a52"
server_key_iv: "210b961e20fd4900d3099235adf34bc7"
server_key_version: "encryptionkey@0"
I20250109 21:08:43.850674 30990 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "dcfb1f7225bc449db7390c0d23854b1c"
format_stamp: "Formatted at 2025-01-09 21:08:43 on dist-test-slave-g0zb"
server_key: "57da9ce5f71e0eff88092cd304be3a52"
server_key_iv: "210b961e20fd4900d3099235adf34bc7"
server_key_version: "encryptionkey@0"
I20250109 21:08:43.853736 30990 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250109 21:08:43.855858 31005 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:43.856812 30990 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:08:43.856923 30990 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "dcfb1f7225bc449db7390c0d23854b1c"
format_stamp: "Formatted at 2025-01-09 21:08:43 on dist-test-slave-g0zb"
server_key: "57da9ce5f71e0eff88092cd304be3a52"
server_key_iv: "210b961e20fd4900d3099235adf34bc7"
server_key_version: "encryptionkey@0"
I20250109 21:08:43.857015 30990 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:43.870208 30990 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:43.872629 30990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:43.872797 30990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:43.873261 30990 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:43.873987 30990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:43.874035 30990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:43.874094 30990 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:43.874145 30990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:43.882633 31118 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:39011 every 8 connection(s)
I20250109 21:08:43.882723 30990 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:39011
I20250109 21:08:43.883600 30990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
Jan 09 21:08:43 dist-test-slave-g0zb krb5kdc[30883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456923, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:08:43.892948 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 30990
I20250109 21:08:43.893070 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:08:43.893369 12793 external_mini_cluster.cc:1453] Setting key 7df0b6cfdd3424d5a22306f92e941078
I20250109 21:08:43.894819 30967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:43.885136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:47339 (local address 127.12.126.126:37259)
0109 21:08:43.885319 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:43.885324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:43.885971 (+ 647us) server_negotiation.cc:407] Connection header received
0109 21:08:43.887192 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:43.887195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:43.887248 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:43.887333 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:43.888488 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:43.888978 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:43.889525 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:43.889653 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:43.891713 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:43.891731 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:43.891733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:43.891754 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:43.892938 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:43.893491 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:43.893493 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:43.893494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:43.893536 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:43.893821 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:43.893823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:43.893825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:43.893992 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:43.894220 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:43.894517 (+ 297us) server_negotiation.cc:299] Negotiation successful
0109 21:08:43.894643 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20250109 21:08:43.895387 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:43.885415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:37259 (local address 127.12.126.65:47339)
0109 21:08:43.885814 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:43.885862 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:43.886992 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:43.887458 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:43.887466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:43.887833 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:43.888378 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:43.888390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:43.889065 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:43.889067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:43.889446 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:43.889452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:43.889606 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:43.890174 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:08:43.890197 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:43.891584 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:43.893057 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:43.893063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:43.893073 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:43.893381 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:43.893616 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:43.893619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:43.893621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:43.893727 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:43.894101 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:43.894109 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:43.894309 (+ 200us) client_negotiation.cc:769] Sending connection context
0109 21:08:43.894459 (+ 150us) client_negotiation.cc:240] Negotiation successful
0109 21:08:43.894664 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":99,"threads_started":1}
I20250109 21:08:43.896394 31119 heartbeater.cc:346] Connected to a master server at 127.12.126.126:37259
I20250109 21:08:43.896601 31119 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:43.897084 31119 heartbeater.cc:510] Master 127.12.126.126:37259 requested a full tablet report, sending...
I20250109 21:08:43.898408 30928 ts_manager.cc:194] Registered new tserver with Master: dcfb1f7225bc449db7390c0d23854b1c (127.12.126.65:39011)
I20250109 21:08:43.899510 30928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:47339
WARNING: no policy specified for kudu/127.12.126.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
I20250109 21:08:43.917340 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:37259
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:44.018513 31124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:44.018718 31124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:44.018771 31124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:44.021797 31124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:44.021852 31124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:44.021924 31124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:08:44.025988 31124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:37259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:44.026968 31124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:44.027736 31124 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:44.035694 31130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:44.035878 31129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:44.036521 31132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:44.037309 31124 server_base.cc:1034] running on GCE node
I20250109 21:08:44.037609 31124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:44.038082 31124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:44.039218 31124 hybrid_clock.cc:648] HybridClock initialized: now 1736456924039204 us; error 36 us; skew 500 ppm
Jan 09 21:08:44 dist-test-slave-g0zb krb5kdc[30883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456924, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:44.041656 31124 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:08:44.042496 31124 webserver.cc:458] Webserver started at http://127.12.126.66:39267/ using document root <none> and password file <none>
I20250109 21:08:44.043051 31124 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:44.043107 31124 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:44.043264 31124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:44.044967 31124 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "a2c5a0b83df94369815f7a4cb5420699"
format_stamp: "Formatted at 2025-01-09 21:08:44 on dist-test-slave-g0zb"
server_key: "55958e8f4b36b332794ab992391f77b1"
server_key_iv: "cf0c043e1f78d8ed8e59fb3631298dcd"
server_key_version: "encryptionkey@0"
I20250109 21:08:44.045431 31124 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "a2c5a0b83df94369815f7a4cb5420699"
format_stamp: "Formatted at 2025-01-09 21:08:44 on dist-test-slave-g0zb"
server_key: "55958e8f4b36b332794ab992391f77b1"
server_key_iv: "cf0c043e1f78d8ed8e59fb3631298dcd"
server_key_version: "encryptionkey@0"
I20250109 21:08:44.048605 31124 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:08:44.050756 31139 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:44.051712 31124 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:44.051821 31124 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "a2c5a0b83df94369815f7a4cb5420699"
format_stamp: "Formatted at 2025-01-09 21:08:44 on dist-test-slave-g0zb"
server_key: "55958e8f4b36b332794ab992391f77b1"
server_key_iv: "cf0c043e1f78d8ed8e59fb3631298dcd"
server_key_version: "encryptionkey@0"
I20250109 21:08:44.051903 31124 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:44.072724 31124 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:44.075333 31124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:44.075481 31124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:44.075953 31124 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:44.076678 31124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:44.076726 31124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:44.076779 31124 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:44.076817 31124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:44.084702 31124 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:36173
I20250109 21:08:44.084753 31252 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:36173 every 8 connection(s)
I20250109 21:08:44.085537 31124 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:08:44.091930 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 31124
I20250109 21:08:44.092044 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:08:44.092307 12793 external_mini_cluster.cc:1453] Setting key 7fbfa4a5611c9918536093b813355d9b
Jan 09 21:08:44 dist-test-slave-g0zb krb5kdc[30883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456924, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:08:44.097698 30967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.087229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:50275 (local address 127.12.126.126:37259)
0109 21:08:44.087370 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:44.087374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:44.087961 (+ 587us) server_negotiation.cc:407] Connection header received
0109 21:08:44.089130 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:44.089134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:44.089196 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:44.089277 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:44.090429 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.090974 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.091577 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.091769 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.093852 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:44.093872 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:44.093876 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:44.093908 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:44.095668 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:44.096283 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:44.096285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:44.096287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:44.096331 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:44.096651 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:44.096654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:44.096655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:44.096794 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:44.096882 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:44.097489 (+ 607us) server_negotiation.cc:299] Negotiation successful
0109 21:08:44.097583 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250109 21:08:44.098510 31255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.087447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:37259 (local address 127.12.126.66:50275)
0109 21:08:44.087833 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:44.087864 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:44.088923 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:44.089436 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:44.089444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:44.089771 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:44.090289 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:44.090301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.091084 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.091087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:44.091469 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:44.091476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.091675 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.092289 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:08:44.092312 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:44.093706 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:44.095792 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:44.095799 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:44.095812 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:44.096179 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:44.096414 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:44.096420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:44.096425 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:44.096539 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:44.096898 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:44.096906 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:44.097209 (+ 303us) client_negotiation.cc:769] Sending connection context
0109 21:08:44.097440 (+ 231us) client_negotiation.cc:240] Negotiation successful
0109 21:08:44.097722 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":97,"threads_started":1}
I20250109 21:08:44.099589 31253 heartbeater.cc:346] Connected to a master server at 127.12.126.126:37259
I20250109 21:08:44.099787 31253 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:44.100224 31253 heartbeater.cc:510] Master 127.12.126.126:37259 requested a full tablet report, sending...
I20250109 21:08:44.101178 30928 ts_manager.cc:194] Registered new tserver with Master: a2c5a0b83df94369815f7a4cb5420699 (127.12.126.66:36173)
WARNING: no policy specified for kudu/127.12.126.67@KRBTEST.COM; defaulting to no policy
I20250109 21:08:44.101765 30928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:50275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
I20250109 21:08:44.118427 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:37259
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:44.217475 31258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:44.217661 31258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:44.217725 31258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:44.220760 31258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:44.220813 31258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:44.220917 31258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:08:44.224884 31258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:37259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:44.225744 31258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:44.226480 31258 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:44.234385 31266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:44.234512 31264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:44.235167 31263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:44.235958 31258 server_base.cc:1034] running on GCE node
I20250109 21:08:44.236253 31258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:44.236686 31258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:44.237823 31258 hybrid_clock.cc:648] HybridClock initialized: now 1736456924237805 us; error 30 us; skew 500 ppm
Jan 09 21:08:44 dist-test-slave-g0zb krb5kdc[30883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456924, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:44.240286 31258 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:08:44.241097 31258 webserver.cc:458] Webserver started at http://127.12.126.67:37547/ using document root <none> and password file <none>
I20250109 21:08:44.241580 31258 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:44.241632 31258 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:44.241816 31258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:44.243465 31258 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "15e29445c9584ba799eec82294ec279f"
format_stamp: "Formatted at 2025-01-09 21:08:44 on dist-test-slave-g0zb"
server_key: "a91d2074b42599bab9308a287690aa1c"
server_key_iv: "247024eb68dda289cc03972606140c79"
server_key_version: "encryptionkey@0"
I20250109 21:08:44.243897 31258 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "15e29445c9584ba799eec82294ec279f"
format_stamp: "Formatted at 2025-01-09 21:08:44 on dist-test-slave-g0zb"
server_key: "a91d2074b42599bab9308a287690aa1c"
server_key_iv: "247024eb68dda289cc03972606140c79"
server_key_version: "encryptionkey@0"
I20250109 21:08:44.246846 31258 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:08:44.248850 31273 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:44.249787 31258 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:44.249886 31258 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "15e29445c9584ba799eec82294ec279f"
format_stamp: "Formatted at 2025-01-09 21:08:44 on dist-test-slave-g0zb"
server_key: "a91d2074b42599bab9308a287690aa1c"
server_key_iv: "247024eb68dda289cc03972606140c79"
server_key_version: "encryptionkey@0"
I20250109 21:08:44.249969 31258 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:44.260774 31258 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:44.263341 31258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:44.263486 31258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:44.263935 31258 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:44.264632 31258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:44.264675 31258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:44.264717 31258 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:44.264752 31258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:44.273015 31258 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:44323
I20250109 21:08:44.273044 31386 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:44323 every 8 connection(s)
I20250109 21:08:44.273833 31258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
Jan 09 21:08:44 dist-test-slave-g0zb krb5kdc[30883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456924, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:08:44.282881 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 31258
I20250109 21:08:44.283000 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:08:44.283252 12793 external_mini_cluster.cc:1453] Setting key 83370a5e9e0fb390931aa0025cba8036
I20250109 21:08:44.284696 30967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.275337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:35013 (local address 127.12.126.126:37259)
0109 21:08:44.275463 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:44.275466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:44.276052 (+ 586us) server_negotiation.cc:407] Connection header received
0109 21:08:44.277159 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:44.277162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:44.277213 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:44.277287 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:44.278356 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.278875 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.279419 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.279546 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.281442 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:44.281459 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:44.281461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:44.281483 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:44.282859 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:44.283365 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:44.283370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:44.283372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:44.283423 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:44.283722 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:44.283724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:44.283726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:44.283895 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:44.283982 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:44.284471 (+ 489us) server_negotiation.cc:299] Negotiation successful
0109 21:08:44.284584 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250109 21:08:44.285326 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.275549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:37259 (local address 127.12.126.67:35013)
0109 21:08:44.275923 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:44.275955 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:44.276963 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:44.277415 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:44.277422 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:44.277740 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:44.278243 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:44.278254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.278967 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.278969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:44.279333 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:44.279340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.279493 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.279956 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:08:44.279975 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:44.281292 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:44.283003 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:44.283011 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:44.283022 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:44.283260 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:44.283515 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:44.283519 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:44.283521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:44.283630 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:44.283990 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:44.283995 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:44.284238 (+ 243us) client_negotiation.cc:769] Sending connection context
0109 21:08:44.284420 (+ 182us) client_negotiation.cc:240] Negotiation successful
0109 21:08:44.284655 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":107,"threads_started":1}
I20250109 21:08:44.286223 31387 heartbeater.cc:346] Connected to a master server at 127.12.126.126:37259
I20250109 21:08:44.286408 31387 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:44.286859 31387 heartbeater.cc:510] Master 127.12.126.126:37259 requested a full tablet report, sending...
I20250109 21:08:44.287796 30928 ts_manager.cc:194] Registered new tserver with Master: 15e29445c9584ba799eec82294ec279f (127.12.126.67:44323)
I20250109 21:08:44.288316 30928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:35013
I20250109 21:08:44.296926 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 21:08:44.304868 30967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.298637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59214 (local address 127.12.126.126:37259)
0109 21:08:44.298787 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:44.298791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:44.298951 (+ 160us) server_negotiation.cc:407] Connection header received
0109 21:08:44.299055 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:44.299057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:44.299097 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:44.299165 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:44.299872 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.300323 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.300852 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.300990 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.302106 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:44.302121 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:44.302123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:44.302144 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:44.303497 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:44.303902 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:44.303904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:44.303905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:44.303941 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:44.304199 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:44.304202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:44.304203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:44.304346 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:44.304465 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:44.304678 (+ 213us) server_negotiation.cc:299] Negotiation successful
0109 21:08:44.304755 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20250109 21:08:44.307365 30928 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:08:44.309696 30928 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:08:44.321955 31399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.317569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:36173 (local address 127.0.0.1:52470)
0109 21:08:44.318379 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:44.318406 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:44.318835 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:44.319344 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:44.319348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:44.319368 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:44.319537 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:44.319542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.320655 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.320659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:44.321462 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:44.321473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.321580 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.321625 (+ 45us) client_negotiation.cc:769] Sending connection context
0109 21:08:44.321733 (+ 108us) client_negotiation.cc:240] Negotiation successful
0109 21:08:44.321823 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":74,"threads_started":1}
I20250109 21:08:44.322240 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.317381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:44323 (local address 127.0.0.1:60678)
0109 21:08:44.317940 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:44.318697 (+ 757us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:44.318848 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:44.319474 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:44.319477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:44.319495 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:44.319712 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:44.319719 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.320985 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.320988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:44.321861 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:44.321872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.321975 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.321992 (+ 17us) client_negotiation.cc:769] Sending connection context
0109 21:08:44.322045 (+ 53us) client_negotiation.cc:240] Negotiation successful
0109 21:08:44.322104 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":74,"threads_started":1}
I20250109 21:08:44.322566 31402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.318044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.12.126.66:36173)
0109 21:08:44.318349 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:44.318355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:44.318485 (+ 130us) server_negotiation.cc:407] Connection header received
0109 21:08:44.318991 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:44.318996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:44.319144 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:44.319262 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:44.319697 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.320528 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.321585 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.322172 (+ 587us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.322288 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:44.322357 (+ 69us) server_negotiation.cc:299] Negotiation successful
0109 21:08:44.322439 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":108,"threads_started":1}
I20250109 21:08:44.323143 31403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.318159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.12.126.65:39011)
0109 21:08:44.318519 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:44.318527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:44.318562 (+ 35us) server_negotiation.cc:407] Connection header received
0109 21:08:44.318855 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:44.318862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:44.319031 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:44.319187 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:44.319621 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.320744 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.322184 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.322721 (+ 537us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.322853 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:44.322921 (+ 68us) server_negotiation.cc:299] Negotiation successful
0109 21:08:44.323003 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":120,"threads_started":1}
I20250109 21:08:44.323247 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.317505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.12.126.67:44323)
0109 21:08:44.318681 (+ 1176us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:44.318688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:44.318975 (+ 287us) server_negotiation.cc:407] Connection header received
0109 21:08:44.319050 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:44.319053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:44.319202 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:44.319357 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:44.319830 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.320878 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.322411 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.322887 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.322979 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:44.323034 (+ 55us) server_negotiation.cc:299] Negotiation successful
0109 21:08:44.323103 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1079,"thread_start_us":130,"threads_started":1}
I20250109 21:08:44.323480 31401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.317983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:39011 (local address 127.0.0.1:47190)
0109 21:08:44.318482 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:44.318496 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:44.318626 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:44.319195 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:44.319200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:44.319222 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:44.319494 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:44.319502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.320898 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.320904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:44.321861 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:44.321872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.321975 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.321992 (+ 17us) client_negotiation.cc:769] Sending connection context
0109 21:08:44.322054 (+ 62us) client_negotiation.cc:240] Negotiation successful
0109 21:08:44.322104 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":85,"threads_started":1}
I20250109 21:08:44.325109 31187 tablet_service.cc:1467] Processing CreateTablet for tablet 255c1fcfa1264762a7eeee09f9227934 (DEFAULT_TABLE table=test-table [id=b2164550196c421997466e8a67691bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:44.325361 31321 tablet_service.cc:1467] Processing CreateTablet for tablet 255c1fcfa1264762a7eeee09f9227934 (DEFAULT_TABLE table=test-table [id=b2164550196c421997466e8a67691bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:44.325974 31187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 255c1fcfa1264762a7eeee09f9227934. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:44.326197 31321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 255c1fcfa1264762a7eeee09f9227934. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:44.325939 31053 tablet_service.cc:1467] Processing CreateTablet for tablet 255c1fcfa1264762a7eeee09f9227934 (DEFAULT_TABLE table=test-table [id=b2164550196c421997466e8a67691bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:44.326704 31053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 255c1fcfa1264762a7eeee09f9227934. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:44.331113 31404 tablet_bootstrap.cc:492] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699: Bootstrap starting.
I20250109 21:08:44.332710 31406 tablet_bootstrap.cc:492] T 255c1fcfa1264762a7eeee09f9227934 P dcfb1f7225bc449db7390c0d23854b1c: Bootstrap starting.
I20250109 21:08:44.333012 31405 tablet_bootstrap.cc:492] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f: Bootstrap starting.
I20250109 21:08:44.333169 31404 tablet_bootstrap.cc:654] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:44.334174 31404 log.cc:826] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:44.335065 31405 tablet_bootstrap.cc:654] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:44.336002 31406 tablet_bootstrap.cc:654] T 255c1fcfa1264762a7eeee09f9227934 P dcfb1f7225bc449db7390c0d23854b1c: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:44.336218 31404 tablet_bootstrap.cc:492] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699: No bootstrap required, opened a new log
I20250109 21:08:44.336252 31405 log.cc:826] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:44.336427 31404 ts_tablet_manager.cc:1397] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:08:44.337020 31406 log.cc:826] T 255c1fcfa1264762a7eeee09f9227934 P dcfb1f7225bc449db7390c0d23854b1c: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:44.338021 31405 tablet_bootstrap.cc:492] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f: No bootstrap required, opened a new log
I20250109 21:08:44.338189 31405 ts_tablet_manager.cc:1397] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:08:44.338382 31406 tablet_bootstrap.cc:492] T 255c1fcfa1264762a7eeee09f9227934 P dcfb1f7225bc449db7390c0d23854b1c: No bootstrap required, opened a new log
I20250109 21:08:44.338528 31406 ts_tablet_manager.cc:1397] T 255c1fcfa1264762a7eeee09f9227934 P dcfb1f7225bc449db7390c0d23854b1c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250109 21:08:44.339664 31404 raft_consensus.cc:357] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e29445c9584ba799eec82294ec279f" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44323 } } peers { permanent_uuid: "a2c5a0b83df94369815f7a4cb5420699" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36173 } } peers { permanent_uuid: "dcfb1f7225bc449db7390c0d23854b1c" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39011 } }
I20250109 21:08:44.339879 31404 raft_consensus.cc:383] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:44.339962 31404 raft_consensus.cc:738] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c5a0b83df94369815f7a4cb5420699, State: Initialized, Role: FOLLOWER
I20250109 21:08:44.340385 31404 consensus_queue.cc:260] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e29445c9584ba799eec82294ec279f" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44323 } } peers { permanent_uuid: "a2c5a0b83df94369815f7a4cb5420699" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36173 } } peers { permanent_uuid: "dcfb1f7225bc449db7390c0d23854b1c" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39011 } }
I20250109 21:08:44.341068 31253 heartbeater.cc:502] Master 127.12.126.126:37259 was elected leader, sending a full tablet report...
I20250109 21:08:44.341260 31404 ts_tablet_manager.cc:1428] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:08:44.341274 31406 raft_consensus.cc:357] T 255c1fcfa1264762a7eeee09f9227934 P dcfb1f7225bc449db7390c0d23854b1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e29445c9584ba799eec82294ec279f" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44323 } } peers { permanent_uuid: "a2c5a0b83df94369815f7a4cb5420699" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36173 } } peers { permanent_uuid: "dcfb1f7225bc449db7390c0d23854b1c" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39011 } }
I20250109 21:08:44.341454 31406 raft_consensus.cc:383] T 255c1fcfa1264762a7eeee09f9227934 P dcfb1f7225bc449db7390c0d23854b1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:44.341513 31406 raft_consensus.cc:738] T 255c1fcfa1264762a7eeee09f9227934 P dcfb1f7225bc449db7390c0d23854b1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfb1f7225bc449db7390c0d23854b1c, State: Initialized, Role: FOLLOWER
I20250109 21:08:44.341892 31406 consensus_queue.cc:260] T 255c1fcfa1264762a7eeee09f9227934 P dcfb1f7225bc449db7390c0d23854b1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e29445c9584ba799eec82294ec279f" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44323 } } peers { permanent_uuid: "a2c5a0b83df94369815f7a4cb5420699" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36173 } } peers { permanent_uuid: "dcfb1f7225bc449db7390c0d23854b1c" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39011 } }
I20250109 21:08:44.342443 31119 heartbeater.cc:502] Master 127.12.126.126:37259 was elected leader, sending a full tablet report...
I20250109 21:08:44.342602 31406 ts_tablet_manager.cc:1428] T 255c1fcfa1264762a7eeee09f9227934 P dcfb1f7225bc449db7390c0d23854b1c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250109 21:08:44.342275 31405 raft_consensus.cc:357] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e29445c9584ba799eec82294ec279f" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44323 } } peers { permanent_uuid: "a2c5a0b83df94369815f7a4cb5420699" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36173 } } peers { permanent_uuid: "dcfb1f7225bc449db7390c0d23854b1c" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39011 } }
I20250109 21:08:44.342854 31405 raft_consensus.cc:383] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:44.342926 31405 raft_consensus.cc:738] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e29445c9584ba799eec82294ec279f, State: Initialized, Role: FOLLOWER
I20250109 21:08:44.343305 31405 consensus_queue.cc:260] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e29445c9584ba799eec82294ec279f" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44323 } } peers { permanent_uuid: "a2c5a0b83df94369815f7a4cb5420699" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36173 } } peers { permanent_uuid: "dcfb1f7225bc449db7390c0d23854b1c" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39011 } }
I20250109 21:08:44.344101 31387 heartbeater.cc:502] Master 127.12.126.126:37259 was elected leader, sending a full tablet report...
I20250109 21:08:44.344403 31405 ts_tablet_manager.cc:1428] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250109 21:08:44.347355 31412 raft_consensus.cc:491] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:08:44.347489 31412 raft_consensus.cc:513] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e29445c9584ba799eec82294ec279f" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44323 } } peers { permanent_uuid: "a2c5a0b83df94369815f7a4cb5420699" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36173 } } peers { permanent_uuid: "dcfb1f7225bc449db7390c0d23854b1c" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39011 } }
I20250109 21:08:44.348336 31412 leader_election.cc:290] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2c5a0b83df94369815f7a4cb5420699 (127.12.126.66:36173), dcfb1f7225bc449db7390c0d23854b1c (127.12.126.65:39011)
I20250109 21:08:44.350965 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.348549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:36173 (local address 127.12.126.67:36965)
0109 21:08:44.348826 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:44.348839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:44.348904 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:44.349165 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:44.349168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:44.349187 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:44.349373 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:44.349379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.350065 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.350068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:44.350596 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:44.350603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.350706 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.350726 (+ 20us) client_negotiation.cc:769] Sending connection context
0109 21:08:44.350777 (+ 51us) client_negotiation.cc:240] Negotiation successful
0109 21:08:44.350852 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212}
I20250109 21:08:44.351128 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.348568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:39011 (local address 127.12.126.67:59325)
0109 21:08:44.349055 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:44.349064 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:44.349129 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:44.349332 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:44.349334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:44.349350 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:44.349561 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:44.349566 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.350198 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.350201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:44.350736 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:44.350743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.350874 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.350891 (+ 17us) client_negotiation.cc:769] Sending connection context
0109 21:08:44.350941 (+ 50us) client_negotiation.cc:240] Negotiation successful
0109 21:08:44.350992 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":95,"threads_started":1}
I20250109 21:08:44.351486 31402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.348745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:36965 (local address 127.12.126.66:36173)
0109 21:08:44.348854 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:44.348857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:44.348870 (+ 13us) server_negotiation.cc:407] Connection header received
0109 21:08:44.349012 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:44.349015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:44.349052 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:44.349136 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:44.349473 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.349959 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.350705 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.351298 (+ 593us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.351329 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:44.351367 (+ 38us) server_negotiation.cc:299] Negotiation successful
0109 21:08:44.351404 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20250109 21:08:44.351550 31403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.348659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:59325 (local address 127.12.126.65:39011)
0109 21:08:44.348783 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:44.348787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:44.349122 (+ 335us) server_negotiation.cc:407] Connection header received
0109 21:08:44.349222 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:44.349224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:44.349257 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:44.349318 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:44.349657 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.350099 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.350858 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.351336 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.351378 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:44.351426 (+ 48us) server_negotiation.cc:299] Negotiation successful
0109 21:08:44.351469 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250109 21:08:44.351945 31207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "255c1fcfa1264762a7eeee09f9227934" candidate_uuid: "15e29445c9584ba799eec82294ec279f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c5a0b83df94369815f7a4cb5420699" is_pre_election: true
I20250109 21:08:44.352013 31073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "255c1fcfa1264762a7eeee09f9227934" candidate_uuid: "15e29445c9584ba799eec82294ec279f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfb1f7225bc449db7390c0d23854b1c" is_pre_election: true
I20250109 21:08:44.352161 31207 raft_consensus.cc:2463] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e29445c9584ba799eec82294ec279f in term 0.
I20250109 21:08:44.352250 31073 raft_consensus.cc:2463] T 255c1fcfa1264762a7eeee09f9227934 P dcfb1f7225bc449db7390c0d23854b1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e29445c9584ba799eec82294ec279f in term 0.
I20250109 21:08:44.352562 31274 leader_election.cc:304] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e29445c9584ba799eec82294ec279f, a2c5a0b83df94369815f7a4cb5420699; no voters:
I20250109 21:08:44.352797 31412 raft_consensus.cc:2798] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:08:44.352880 31412 raft_consensus.cc:491] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:08:44.352931 31412 raft_consensus.cc:3054] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:44.353736 31412 raft_consensus.cc:513] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e29445c9584ba799eec82294ec279f" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44323 } } peers { permanent_uuid: "a2c5a0b83df94369815f7a4cb5420699" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36173 } } peers { permanent_uuid: "dcfb1f7225bc449db7390c0d23854b1c" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39011 } }
I20250109 21:08:44.354104 31412 leader_election.cc:290] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [CANDIDATE]: Term 1 election: Requested vote from peers a2c5a0b83df94369815f7a4cb5420699 (127.12.126.66:36173), dcfb1f7225bc449db7390c0d23854b1c (127.12.126.65:39011)
I20250109 21:08:44.354400 31207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "255c1fcfa1264762a7eeee09f9227934" candidate_uuid: "15e29445c9584ba799eec82294ec279f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c5a0b83df94369815f7a4cb5420699"
I20250109 21:08:44.354508 31207 raft_consensus.cc:3054] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:44.354568 31073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "255c1fcfa1264762a7eeee09f9227934" candidate_uuid: "15e29445c9584ba799eec82294ec279f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfb1f7225bc449db7390c0d23854b1c"
I20250109 21:08:44.354658 31073 raft_consensus.cc:3054] T 255c1fcfa1264762a7eeee09f9227934 P dcfb1f7225bc449db7390c0d23854b1c [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:44.355743 31207 raft_consensus.cc:2463] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e29445c9584ba799eec82294ec279f in term 1.
I20250109 21:08:44.355798 31073 raft_consensus.cc:2463] T 255c1fcfa1264762a7eeee09f9227934 P dcfb1f7225bc449db7390c0d23854b1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e29445c9584ba799eec82294ec279f in term 1.
I20250109 21:08:44.356081 31275 leader_election.cc:304] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e29445c9584ba799eec82294ec279f, dcfb1f7225bc449db7390c0d23854b1c; no voters:
I20250109 21:08:44.356235 31412 raft_consensus.cc:2798] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:44.356472 31412 raft_consensus.cc:695] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [term 1 LEADER]: Becoming Leader. State: Replica: 15e29445c9584ba799eec82294ec279f, State: Running, Role: LEADER
I20250109 21:08:44.356760 31412 consensus_queue.cc:237] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e29445c9584ba799eec82294ec279f" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44323 } } peers { permanent_uuid: "a2c5a0b83df94369815f7a4cb5420699" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36173 } } peers { permanent_uuid: "dcfb1f7225bc449db7390c0d23854b1c" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39011 } }
I20250109 21:08:44.359308 30926 catalog_manager.cc:5526] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f reported cstate change: term changed from 0 to 1, leader changed from <none> to 15e29445c9584ba799eec82294ec279f (127.12.126.67). New cstate: current_term: 1 leader_uuid: "15e29445c9584ba799eec82294ec279f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e29445c9584ba799eec82294ec279f" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 44323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c5a0b83df94369815f7a4cb5420699" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfb1f7225bc449db7390c0d23854b1c" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39011 } health_report { overall_health: UNKNOWN } } }
I20250109 21:08:44.371214 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.368374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.12.126.67:44323)
0109 21:08:44.368488 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:44.368491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:44.368503 (+ 12us) server_negotiation.cc:407] Connection header received
0109 21:08:44.368572 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:44.368574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:44.368630 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:44.368708 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:08:44.369148 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.369604 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.370201 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.370332 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.370525 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:08:44.370907 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:08:44.371023 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:44.371105 (+ 82us) server_negotiation.cc:299] Negotiation successful
0109 21:08:44.371130 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250109 21:08:44.375648 31073 raft_consensus.cc:1270] T 255c1fcfa1264762a7eeee09f9227934 P dcfb1f7225bc449db7390c0d23854b1c [term 1 FOLLOWER]: Refusing update from remote peer 15e29445c9584ba799eec82294ec279f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:08:44.375655 31207 raft_consensus.cc:1270] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699 [term 1 FOLLOWER]: Refusing update from remote peer 15e29445c9584ba799eec82294ec279f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:08:44.376219 31414 consensus_queue.cc:1035] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfb1f7225bc449db7390c0d23854b1c" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 39011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:08:44.376367 31412 consensus_queue.cc:1035] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c5a0b83df94369815f7a4cb5420699" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 36173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:08:44.383738 31418 mvcc.cc:204] Tried to move back new op lower bound from 7112527562237083648 to 7112527562167660544. Current Snapshot: MvccSnapshot[applied={T|T < 7112527562237083648}]
I20250109 21:08:44.383981 31419 mvcc.cc:204] Tried to move back new op lower bound from 7112527562237083648 to 7112527562167660544. Current Snapshot: MvccSnapshot[applied={T|T < 7112527562237083648}]
I20250109 21:08:44.389698 30926 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59214:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:08:44.389854 30926 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59214:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:08:44.392031 30926 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 7946cf4ac44149088877f35dab850653: Sending DeleteTablet for 3 replicas of tablet 255c1fcfa1264762a7eeee09f9227934
I20250109 21:08:44.392705 31187 tablet_service.cc:1514] Processing DeleteTablet for tablet 255c1fcfa1264762a7eeee09f9227934 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:44 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:52470
I20250109 21:08:44.392740 31053 tablet_service.cc:1514] Processing DeleteTablet for tablet 255c1fcfa1264762a7eeee09f9227934 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:44 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:47190
I20250109 21:08:44.393148 31426 tablet_replica.cc:331] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699: stopping tablet replica
I20250109 21:08:44.393369 31321 tablet_service.cc:1514] Processing DeleteTablet for tablet 255c1fcfa1264762a7eeee09f9227934 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:44 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:60678
I20250109 21:08:44.393584 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 30990
I20250109 21:08:44.395655 31426 raft_consensus.cc:2238] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:08:44.395784 31428 tablet_replica.cc:331] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f: stopping tablet replica
I20250109 21:08:44.395951 31426 raft_consensus.cc:2267] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:44.396023 31428 raft_consensus.cc:2238] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:08:44.396345 31428 raft_consensus.cc:2267] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:44.397136 31426 ts_tablet_manager.cc:1905] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:08:44.399127 31426 ts_tablet_manager.cc:1918] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:08:44.399207 31426 log.cc:1198] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/255c1fcfa1264762a7eeee09f9227934
I20250109 21:08:44.399533 31426 ts_tablet_manager.cc:1939] T 255c1fcfa1264762a7eeee09f9227934 P a2c5a0b83df94369815f7a4cb5420699: Deleting consensus metadata
I20250109 21:08:44.400475 30912 catalog_manager.cc:4872] TS a2c5a0b83df94369815f7a4cb5420699 (127.12.126.66:36173): tablet 255c1fcfa1264762a7eeee09f9227934 (table test-table [id=b2164550196c421997466e8a67691bf4]) successfully deleted
W20250109 21:08:44.402400 30913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.126.65:39011 (error 108)
I20250109 21:08:44.403260 31428 ts_tablet_manager.cc:1905] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:08:44.403304 31401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.403073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:39011 (local address 127.0.0.1:47200)
0109 21:08:44.403189 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:44.403241 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.65:39011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20250109 21:08:44.403407 30913 catalog_manager.cc:4615] TS dcfb1f7225bc449db7390c0d23854b1c (127.12.126.65:39011): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 255c1fcfa1264762a7eeee09f9227934: Network error: Client connection negotiation failed: client connection to 127.12.126.65:39011: connect: Connection refused (error 111)
I20250109 21:08:44.405709 31428 ts_tablet_manager.cc:1918] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:08:44.405817 31428 log.cc:1198] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/255c1fcfa1264762a7eeee09f9227934
I20250109 21:08:44.405961 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 31124
I20250109 21:08:44.410730 31428 ts_tablet_manager.cc:1939] T 255c1fcfa1264762a7eeee09f9227934 P 15e29445c9584ba799eec82294ec279f: Deleting consensus metadata
I20250109 21:08:44.410918 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 31258
I20250109 21:08:44.416355 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 30896
2025-01-09T21:08:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2078 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2025-01-09T21:08:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:08:44Z Disabled control of system clock
I20250109 21:08:44.444339 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:46267
--webserver_interface=127.12.126.126
--webserver_port=0
--builtin_ntp_servers=127.12.126.84:40389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:46267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20250109 21:08:44.541085 31433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:44.541260 31433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:44.541301 31433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:44.544198 31433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:08:44.544243 31433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:44.544261 31433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:44.544279 31433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:08:44.544296 31433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:08:44.547608 31433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:40389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:46267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:46267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:44.548491 31433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:44.549439 31433 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:44.555817 31439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:44.556414 31441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:44.556938 31438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:44.557709 31433 server_base.cc:1034] running on GCE node
I20250109 21:08:44.558099 31433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:44.558851 31433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:44.560009 31433 hybrid_clock.cc:648] HybridClock initialized: now 1736456924559984 us; error 38 us; skew 500 ppm
I20250109 21:08:44.561733 31433 webserver.cc:458] Webserver started at http://127.12.126.126:38821/ using document root <none> and password file <none>
I20250109 21:08:44.562245 31433 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:44.562299 31433 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:44.562462 31433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:44.564065 31433 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "67bd68d8d4bb48b0b10bd9fbb5540fa2"
format_stamp: "Formatted at 2025-01-09 21:08:44 on dist-test-slave-g0zb"
server_key: "67492a7b4a09b0461fe447ca55731f55"
server_key_iv: "7965f2416cae8ff31251c8d7f02c7e86"
server_key_version: "encryptionkey@0"
I20250109 21:08:44.564499 31433 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "67bd68d8d4bb48b0b10bd9fbb5540fa2"
format_stamp: "Formatted at 2025-01-09 21:08:44 on dist-test-slave-g0zb"
server_key: "67492a7b4a09b0461fe447ca55731f55"
server_key_iv: "7965f2416cae8ff31251c8d7f02c7e86"
server_key_version: "encryptionkey@0"
I20250109 21:08:44.567488 31433 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20250109 21:08:44.569343 31447 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:44.570225 31433 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 21:08:44.570333 31433 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "67bd68d8d4bb48b0b10bd9fbb5540fa2"
format_stamp: "Formatted at 2025-01-09 21:08:44 on dist-test-slave-g0zb"
server_key: "67492a7b4a09b0461fe447ca55731f55"
server_key_iv: "7965f2416cae8ff31251c8d7f02c7e86"
server_key_version: "encryptionkey@0"
I20250109 21:08:44.570408 31433 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:44.587900 31433 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:44.588402 31433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:44.588543 31433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:44.594223 31433 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:46267
I20250109 21:08:44.594239 31499 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:46267 every 8 connection(s)
I20250109 21:08:44.595045 31433 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:08:44.597210 31500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:44.598742 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 31433
I20250109 21:08:44.598886 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:08:44.599149 12793 external_mini_cluster.cc:1453] Setting key 4d63005160239a6c35ce6de07f59357f
I20250109 21:08:44.602653 31500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2: Bootstrap starting.
I20250109 21:08:44.604588 31500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:44.605222 31500 log.cc:826] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:44.605952 31503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.600284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34720 (local address 127.12.126.126:46267)
0109 21:08:44.600795 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:44.600805 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:44.600836 (+ 31us) server_negotiation.cc:407] Connection header received
0109 21:08:44.601351 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:44.601367 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:44.601678 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:44.601987 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:44.602461 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.603241 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.603854 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.604119 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.604580 (+ 461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:44.604596 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:44.604605 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:44.604938 (+ 333us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:44.604974 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:44.604990 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:44.605090 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:44.605225 (+ 135us) server_negotiation.cc:299] Negotiation successful
0109 21:08:44.605409 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":103,"threads_started":1}
I20250109 21:08:44.607057 31500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2: No bootstrap required, opened a new log
I20250109 21:08:44.609725 31500 raft_consensus.cc:357] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bd68d8d4bb48b0b10bd9fbb5540fa2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 46267 } }
I20250109 21:08:44.609879 31500 raft_consensus.cc:383] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:44.609964 31500 raft_consensus.cc:738] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67bd68d8d4bb48b0b10bd9fbb5540fa2, State: Initialized, Role: FOLLOWER
I20250109 21:08:44.610311 31500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bd68d8d4bb48b0b10bd9fbb5540fa2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 46267 } }
I20250109 21:08:44.610424 31500 raft_consensus.cc:397] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:08:44.610482 31500 raft_consensus.cc:491] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:08:44.610595 31500 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:44.611296 31500 raft_consensus.cc:513] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bd68d8d4bb48b0b10bd9fbb5540fa2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 46267 } }
I20250109 21:08:44.611590 31500 leader_election.cc:304] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67bd68d8d4bb48b0b10bd9fbb5540fa2; no voters:
I20250109 21:08:44.611865 31500 leader_election.cc:290] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:08:44.612099 31505 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:44.612344 31505 raft_consensus.cc:695] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2 [term 1 LEADER]: Becoming Leader. State: Replica: 67bd68d8d4bb48b0b10bd9fbb5540fa2, State: Running, Role: LEADER
I20250109 21:08:44.612620 31505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bd68d8d4bb48b0b10bd9fbb5540fa2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 46267 } }
I20250109 21:08:44.612888 31500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:08:44.614037 31507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67bd68d8d4bb48b0b10bd9fbb5540fa2. Latest consensus state: current_term: 1 leader_uuid: "67bd68d8d4bb48b0b10bd9fbb5540fa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bd68d8d4bb48b0b10bd9fbb5540fa2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 46267 } } }
I20250109 21:08:44.614156 31507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:44.614436 31506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67bd68d8d4bb48b0b10bd9fbb5540fa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bd68d8d4bb48b0b10bd9fbb5540fa2" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 46267 } } }
I20250109 21:08:44.614521 31514 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:08:44.614583 31506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:44.616817 31514 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:08:44.621404 31514 catalog_manager.cc:1348] Generated new cluster ID: fa0537d356b44c43bd16b1203770d926
I20250109 21:08:44.621452 31514 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:08:44.641091 31514 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:08:44.642129 31514 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:08:44.650558 31514 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2: Generated new TSK 0
I20250109 21:08:44.651168 31514 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:08:44.663021 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:46267
--builtin_ntp_servers=127.12.126.84:40389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20250109 21:08:44.768630 31524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:44.768828 31524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:44.768900 31524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:44.771879 31524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:44.771929 31524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:44.772022 31524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:08:44.775527 31524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:40389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:46267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:44.776428 31524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:44.777407 31524 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:44.785027 31532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:44.785662 31529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:44.786198 31530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:44.786929 31524 server_base.cc:1034] running on GCE node
I20250109 21:08:44.787205 31524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:44.787675 31524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:44.788815 31524 hybrid_clock.cc:648] HybridClock initialized: now 1736456924788794 us; error 34 us; skew 500 ppm
I20250109 21:08:44.790390 31524 webserver.cc:458] Webserver started at http://127.12.126.65:42015/ using document root <none> and password file <none>
I20250109 21:08:44.790892 31524 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:44.790946 31524 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:44.791115 31524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:44.792703 31524 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "11f19c9f345e48478dc46eff27952481"
format_stamp: "Formatted at 2025-01-09 21:08:44 on dist-test-slave-g0zb"
server_key: "c857ed38bc76f7e46a82abc3007eb902"
server_key_iv: "2a4352b9224078ccf167d33418b10f82"
server_key_version: "encryptionkey@0"
I20250109 21:08:44.793128 31524 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "11f19c9f345e48478dc46eff27952481"
format_stamp: "Formatted at 2025-01-09 21:08:44 on dist-test-slave-g0zb"
server_key: "c857ed38bc76f7e46a82abc3007eb902"
server_key_iv: "2a4352b9224078ccf167d33418b10f82"
server_key_version: "encryptionkey@0"
I20250109 21:08:44.796015 31524 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:08:44.797785 31538 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:44.798667 31524 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:44.798761 31524 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "11f19c9f345e48478dc46eff27952481"
format_stamp: "Formatted at 2025-01-09 21:08:44 on dist-test-slave-g0zb"
server_key: "c857ed38bc76f7e46a82abc3007eb902"
server_key_iv: "2a4352b9224078ccf167d33418b10f82"
server_key_version: "encryptionkey@0"
I20250109 21:08:44.798879 31524 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:44.812024 31524 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:44.812486 31524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:44.812618 31524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:44.813017 31524 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:44.813715 31524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:44.813760 31524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:44.813808 31524 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:44.813840 31524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:44.821942 31524 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:36741
I20250109 21:08:44.821964 31651 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:36741 every 8 connection(s)
I20250109 21:08:44.822746 31524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:08:44.828418 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 31524
I20250109 21:08:44.828536 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:08:44.828814 12793 external_mini_cluster.cc:1453] Setting key e27dc712965cddce40a881e92a549328
I20250109 21:08:44.829562 31503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.824149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:60279 (local address 127.12.126.126:46267)
0109 21:08:44.824277 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:44.824281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:44.825001 (+ 720us) server_negotiation.cc:407] Connection header received
0109 21:08:44.825691 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:44.825694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:44.825753 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:44.825827 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:44.827065 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.827604 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.828180 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.828364 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.829074 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:44.829091 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:44.829093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:44.829103 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:44.829121 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:44.829128 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:44.829185 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:44.829367 (+ 182us) server_negotiation.cc:299] Negotiation successful
0109 21:08:44.829433 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250109 21:08:44.830155 31654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.824380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:46267 (local address 127.12.126.65:60279)
0109 21:08:44.824888 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:44.824913 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:44.825498 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:44.825973 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:44.825980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:44.826048 (+ 68us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:44.826928 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:44.826941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.827716 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.827720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:44.828073 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:44.828080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.828293 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.828850 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:08:44.828864 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:44.828891 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:08:44.828952 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:44.829193 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:44.829202 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:44.829207 (+ 5us) client_negotiation.cc:769] Sending connection context
0109 21:08:44.829347 (+ 140us) client_negotiation.cc:240] Negotiation successful
0109 21:08:44.829520 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20250109 21:08:44.830761 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:46267
--builtin_ntp_servers=127.12.126.84:40389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20250109 21:08:44.831282 31652 heartbeater.cc:346] Connected to a master server at 127.12.126.126:46267
I20250109 21:08:44.831466 31652 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:44.831899 31652 heartbeater.cc:510] Master 127.12.126.126:46267 requested a full tablet report, sending...
I20250109 21:08:44.833187 31464 ts_manager.cc:194] Registered new tserver with Master: 11f19c9f345e48478dc46eff27952481 (127.12.126.65:36741)
I20250109 21:08:44.834182 31464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.126.65:60279
W20250109 21:08:44.926960 31655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:44.927150 31655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:44.927213 31655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:44.930291 31655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:44.930394 31655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:44.930517 31655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:08:44.934556 31655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:40389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:46267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:44.935492 31655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:44.936554 31655 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:44.944295 31661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:44.944941 31663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:44.945537 31660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:44.946362 31655 server_base.cc:1034] running on GCE node
I20250109 21:08:44.946652 31655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:44.947157 31655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:44.948300 31655 hybrid_clock.cc:648] HybridClock initialized: now 1736456924948280 us; error 34 us; skew 500 ppm
I20250109 21:08:44.949952 31655 webserver.cc:458] Webserver started at http://127.12.126.66:35983/ using document root <none> and password file <none>
I20250109 21:08:44.950471 31655 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:44.950520 31655 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:44.950687 31655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:44.952376 31655 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "f3c17bb55527494eab8d8286d9e91ee4"
format_stamp: "Formatted at 2025-01-09 21:08:44 on dist-test-slave-g0zb"
server_key: "c2908faa7acb91b341dfe9678f85774c"
server_key_iv: "67a29820e0ad74ebadc62448453e4104"
server_key_version: "encryptionkey@0"
I20250109 21:08:44.952797 31655 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "f3c17bb55527494eab8d8286d9e91ee4"
format_stamp: "Formatted at 2025-01-09 21:08:44 on dist-test-slave-g0zb"
server_key: "c2908faa7acb91b341dfe9678f85774c"
server_key_iv: "67a29820e0ad74ebadc62448453e4104"
server_key_version: "encryptionkey@0"
I20250109 21:08:44.955772 31655 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20250109 21:08:44.957636 31669 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:44.958541 31655 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 21:08:44.958637 31655 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "f3c17bb55527494eab8d8286d9e91ee4"
format_stamp: "Formatted at 2025-01-09 21:08:44 on dist-test-slave-g0zb"
server_key: "c2908faa7acb91b341dfe9678f85774c"
server_key_iv: "67a29820e0ad74ebadc62448453e4104"
server_key_version: "encryptionkey@0"
I20250109 21:08:44.958703 31655 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:44.976161 31655 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:44.976617 31655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:44.976733 31655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:44.977169 31655 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:44.977928 31655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:44.977972 31655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:44.978013 31655 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:44.978036 31655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:44.985563 31655 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:35705
I20250109 21:08:44.985605 31782 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:35705 every 8 connection(s)
I20250109 21:08:44.986379 31655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:08:44.993270 31503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.988030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:40553 (local address 127.12.126.126:46267)
0109 21:08:44.988178 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:44.988183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:44.989041 (+ 858us) server_negotiation.cc:407] Connection header received
0109 21:08:44.989748 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:44.989751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:44.989809 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:44.989873 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:44.991206 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.991648 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.992151 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.992256 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.992924 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:44.992939 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:44.992941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:44.992952 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:44.992969 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:44.992976 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:44.993031 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:44.993130 (+ 99us) server_negotiation.cc:299] Negotiation successful
0109 21:08:44.993174 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250109 21:08:44.993834 31785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:44.988286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:46267 (local address 127.12.126.66:40553)
0109 21:08:44.988919 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:44.988955 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:44.989559 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:44.990000 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:44.990008 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:44.990098 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:44.991065 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:44.991079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.991731 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:44.991733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:44.992076 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:44.992083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:44.992221 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:44.992735 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:08:44.992737 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:44.992764 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:08:44.992810 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:44.993037 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:44.993041 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:44.993044 (+ 3us) client_negotiation.cc:769] Sending connection context
0109 21:08:44.993126 (+ 82us) client_negotiation.cc:240] Negotiation successful
0109 21:08:44.993271 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":90,"threads_started":1}
I20250109 21:08:44.994707 31783 heartbeater.cc:346] Connected to a master server at 127.12.126.126:46267
I20250109 21:08:44.994901 31783 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:44.995344 31783 heartbeater.cc:510] Master 127.12.126.126:46267 requested a full tablet report, sending...
I20250109 21:08:44.995673 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 31655
I20250109 21:08:44.995791 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:08:44.996031 12793 external_mini_cluster.cc:1453] Setting key e8baa58050e1bb996bf5c34da5af5d66
I20250109 21:08:44.996294 31464 ts_manager.cc:194] Registered new tserver with Master: f3c17bb55527494eab8d8286d9e91ee4 (127.12.126.66:35705)
I20250109 21:08:44.996872 31464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.126.66:40553
I20250109 21:08:44.997970 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:46267
--builtin_ntp_servers=127.12.126.84:40389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20250109 21:08:45.097275 31786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:45.097498 31786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:45.097550 31786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:45.100546 31786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:45.100586 31786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:45.100665 31786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:08:45.104161 31786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:40389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:46267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:45.104998 31786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:45.105973 31786 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:45.113315 31792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:45.113924 31791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:45.114429 31794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:45.115047 31786 server_base.cc:1034] running on GCE node
I20250109 21:08:45.115326 31786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:45.115792 31786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:45.116942 31786 hybrid_clock.cc:648] HybridClock initialized: now 1736456925116927 us; error 27 us; skew 500 ppm
I20250109 21:08:45.118626 31786 webserver.cc:458] Webserver started at http://127.12.126.67:40205/ using document root <none> and password file <none>
I20250109 21:08:45.119170 31786 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:45.119230 31786 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:45.119387 31786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:45.121037 31786 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "c256f6bc46fd470e9c7aa76162f562a8"
format_stamp: "Formatted at 2025-01-09 21:08:45 on dist-test-slave-g0zb"
server_key: "8f15793eb26ef564e677be0aac1eb956"
server_key_iv: "8bafcd0c726104a8a4a1f4e0c8475258"
server_key_version: "encryptionkey@0"
I20250109 21:08:45.121471 31786 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "c256f6bc46fd470e9c7aa76162f562a8"
format_stamp: "Formatted at 2025-01-09 21:08:45 on dist-test-slave-g0zb"
server_key: "8f15793eb26ef564e677be0aac1eb956"
server_key_iv: "8bafcd0c726104a8a4a1f4e0c8475258"
server_key_version: "encryptionkey@0"
I20250109 21:08:45.124650 31786 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:08:45.126497 31800 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:45.127380 31786 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:45.127492 31786 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "c256f6bc46fd470e9c7aa76162f562a8"
format_stamp: "Formatted at 2025-01-09 21:08:45 on dist-test-slave-g0zb"
server_key: "8f15793eb26ef564e677be0aac1eb956"
server_key_iv: "8bafcd0c726104a8a4a1f4e0c8475258"
server_key_version: "encryptionkey@0"
I20250109 21:08:45.127586 31786 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:45.151688 31786 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:45.152199 31786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:45.152350 31786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:45.152783 31786 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:45.153551 31786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:45.153602 31786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:45.153653 31786 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:45.153679 31786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:45.161252 31786 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:41193
I20250109 21:08:45.161280 31913 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:41193 every 8 connection(s)
I20250109 21:08:45.162084 31786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:08:45.162966 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 31786
I20250109 21:08:45.163079 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:08:45.163296 12793 external_mini_cluster.cc:1453] Setting key a53f53149844df4ecc5d94208634937c
I20250109 21:08:45.169121 31503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:45.163689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:48879 (local address 127.12.126.126:46267)
0109 21:08:45.163813 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:45.163820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:45.164714 (+ 894us) server_negotiation.cc:407] Connection header received
0109 21:08:45.165441 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:45.165444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:45.165489 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:45.165554 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:45.166845 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.167323 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:45.167916 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.168037 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:45.168746 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:45.168775 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:45.168777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:45.168789 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:45.168807 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:45.168814 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:45.168871 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:45.168974 (+ 103us) server_negotiation.cc:299] Negotiation successful
0109 21:08:45.169019 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20250109 21:08:45.169684 31916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:45.164017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:46267 (local address 127.12.126.67:48879)
0109 21:08:45.164595 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:45.164632 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:45.165277 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:45.165679 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:45.165686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:45.165759 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:45.166711 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:45.166723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.167435 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:45.167437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:45.167831 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:45.167838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.168001 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:45.168541 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:08:45.168542 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:45.168569 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:08:45.168619 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:45.168876 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:45.168880 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:45.168883 (+ 3us) client_negotiation.cc:769] Sending connection context
0109 21:08:45.168980 (+ 97us) client_negotiation.cc:240] Negotiation successful
0109 21:08:45.169136 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":81,"threads_started":1}
I20250109 21:08:45.170601 31914 heartbeater.cc:346] Connected to a master server at 127.12.126.126:46267
I20250109 21:08:45.170804 31914 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:45.171331 31914 heartbeater.cc:510] Master 127.12.126.126:46267 requested a full tablet report, sending...
I20250109 21:08:45.172230 31463 ts_manager.cc:194] Registered new tserver with Master: c256f6bc46fd470e9c7aa76162f562a8 (127.12.126.67:41193)
I20250109 21:08:45.172768 31463 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.126.67:48879
I20250109 21:08:45.176616 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 21:08:45.181208 31503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:45.178270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34736 (local address 127.12.126.126:46267)
0109 21:08:45.178381 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:45.178384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:45.178500 (+ 116us) server_negotiation.cc:407] Connection header received
0109 21:08:45.178617 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:45.178619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:45.178653 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:45.178716 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:45.179100 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.179534 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:45.180141 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.180252 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:45.180858 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:45.180870 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:45.180872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:45.180882 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:45.180898 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:45.180905 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:45.180989 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:45.181085 (+ 96us) server_negotiation.cc:299] Negotiation successful
0109 21:08:45.181113 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250109 21:08:45.183740 31463 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:08:45.186219 31463 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:08:45.198582 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:45.193877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:35705 (local address 127.0.0.1:60414)
0109 21:08:45.194924 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:45.194962 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:45.195175 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:45.195902 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:45.195908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:45.195942 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:45.196273 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:45.196282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.197251 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:45.197254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:45.198109 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:45.198118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.198218 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:45.198268 (+ 50us) client_negotiation.cc:769] Sending connection context
0109 21:08:45.198362 (+ 94us) client_negotiation.cc:240] Negotiation successful
0109 21:08:45.198443 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"spinlock_wait_cycles":7040,"thread_start_us":80,"threads_started":1}
I20250109 21:08:45.199110 31925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:45.194554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:41193 (local address 127.0.0.1:32954)
0109 21:08:45.195356 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:45.195369 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:45.195451 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:45.195902 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:45.195908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:45.195943 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:45.196273 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:45.196282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.197843 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:45.197847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:45.198822 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:45.198833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.198929 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:45.198942 (+ 13us) client_negotiation.cc:769] Sending connection context
0109 21:08:45.198980 (+ 38us) client_negotiation.cc:240] Negotiation successful
0109 21:08:45.199023 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"spinlock_wait_cycles":8704,"thread_start_us":88,"threads_started":1}
I20250109 21:08:45.199249 31927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:45.194138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60414 (local address 127.12.126.66:35705)
0109 21:08:45.194517 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:45.194524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:45.195058 (+ 534us) server_negotiation.cc:407] Connection header received
0109 21:08:45.195396 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:45.195403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:45.195600 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:45.195732 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:45.196439 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.197143 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:45.198241 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.198749 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:45.198951 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:45.199016 (+ 65us) server_negotiation.cc:299] Negotiation successful
0109 21:08:45.199111 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":134,"threads_started":1}
I20250109 21:08:45.199990 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:45.194965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.12.126.67:41193)
0109 21:08:45.195329 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:45.195337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:45.195427 (+ 90us) server_negotiation.cc:407] Connection header received
0109 21:08:45.195571 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:45.195575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:45.195753 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:45.195897 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:45.196408 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.197726 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:45.199260 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.199648 (+ 388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:45.199750 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:45.199816 (+ 66us) server_negotiation.cc:299] Negotiation successful
0109 21:08:45.199875 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":161,"threads_started":1}
I20250109 21:08:45.200029 31928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:45.194477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34274 (local address 127.12.126.65:36741)
0109 21:08:45.194831 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:45.194837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:45.195107 (+ 270us) server_negotiation.cc:407] Connection header received
0109 21:08:45.195572 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:45.195576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:45.195753 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:45.195875 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:45.196378 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.197539 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:45.198931 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.199407 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:45.199539 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:45.199787 (+ 248us) server_negotiation.cc:299] Negotiation successful
0109 21:08:45.199889 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":123,"threads_started":1}
I20250109 21:08:45.200397 31926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:45.193974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:36741 (local address 127.0.0.1:34274)
0109 21:08:45.195014 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:45.195026 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:45.195386 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:45.196068 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:45.196070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:45.196086 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:45.196261 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:45.196266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.197666 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:45.197669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:45.198750 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:45.198761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.199597 (+ 836us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:45.199612 (+ 15us) client_negotiation.cc:769] Sending connection context
0109 21:08:45.200206 (+ 594us) client_negotiation.cc:240] Negotiation successful
0109 21:08:45.200267 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"thread_start_us":102,"threads_started":1}
I20250109 21:08:45.202292 31717 tablet_service.cc:1467] Processing CreateTablet for tablet 4c602e0fecea4f0abd6d590201aaa264 (DEFAULT_TABLE table=test-table [id=5aed0ba210d94fbeab20a730fe23536a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:45.202641 31848 tablet_service.cc:1467] Processing CreateTablet for tablet 4c602e0fecea4f0abd6d590201aaa264 (DEFAULT_TABLE table=test-table [id=5aed0ba210d94fbeab20a730fe23536a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:45.202685 31586 tablet_service.cc:1467] Processing CreateTablet for tablet 4c602e0fecea4f0abd6d590201aaa264 (DEFAULT_TABLE table=test-table [id=5aed0ba210d94fbeab20a730fe23536a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:45.203357 31717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c602e0fecea4f0abd6d590201aaa264. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:45.203375 31848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c602e0fecea4f0abd6d590201aaa264. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:45.203464 31586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c602e0fecea4f0abd6d590201aaa264. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:45.208633 31931 tablet_bootstrap.cc:492] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8: Bootstrap starting.
I20250109 21:08:45.209918 31933 tablet_bootstrap.cc:492] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4: Bootstrap starting.
I20250109 21:08:45.211126 31932 tablet_bootstrap.cc:492] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481: Bootstrap starting.
I20250109 21:08:45.211172 31931 tablet_bootstrap.cc:654] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:45.211848 31933 tablet_bootstrap.cc:654] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:45.212136 31931 log.cc:826] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:45.213074 31933 log.cc:826] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:45.213244 31932 tablet_bootstrap.cc:654] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:45.213979 31931 tablet_bootstrap.cc:492] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8: No bootstrap required, opened a new log
I20250109 21:08:45.214148 31931 ts_tablet_manager.cc:1397] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250109 21:08:45.214318 31932 log.cc:826] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:45.214926 31933 tablet_bootstrap.cc:492] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4: No bootstrap required, opened a new log
I20250109 21:08:45.215092 31933 ts_tablet_manager.cc:1397] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:08:45.216727 31932 tablet_bootstrap.cc:492] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481: No bootstrap required, opened a new log
I20250109 21:08:45.216913 31932 ts_tablet_manager.cc:1397] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250109 21:08:45.217602 31933 raft_consensus.cc:357] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c17bb55527494eab8d8286d9e91ee4" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 35705 } } peers { permanent_uuid: "11f19c9f345e48478dc46eff27952481" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 36741 } } peers { permanent_uuid: "c256f6bc46fd470e9c7aa76162f562a8" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 41193 } }
I20250109 21:08:45.217584 31931 raft_consensus.cc:357] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c17bb55527494eab8d8286d9e91ee4" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 35705 } } peers { permanent_uuid: "11f19c9f345e48478dc46eff27952481" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 36741 } } peers { permanent_uuid: "c256f6bc46fd470e9c7aa76162f562a8" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 41193 } }
I20250109 21:08:45.217773 31933 raft_consensus.cc:383] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:45.217800 31931 raft_consensus.cc:383] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:45.217831 31933 raft_consensus.cc:738] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c17bb55527494eab8d8286d9e91ee4, State: Initialized, Role: FOLLOWER
I20250109 21:08:45.217850 31931 raft_consensus.cc:738] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c256f6bc46fd470e9c7aa76162f562a8, State: Initialized, Role: FOLLOWER
I20250109 21:08:45.218230 31933 consensus_queue.cc:260] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c17bb55527494eab8d8286d9e91ee4" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 35705 } } peers { permanent_uuid: "11f19c9f345e48478dc46eff27952481" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 36741 } } peers { permanent_uuid: "c256f6bc46fd470e9c7aa76162f562a8" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 41193 } }
I20250109 21:08:45.218286 31931 consensus_queue.cc:260] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c17bb55527494eab8d8286d9e91ee4" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 35705 } } peers { permanent_uuid: "11f19c9f345e48478dc46eff27952481" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 36741 } } peers { permanent_uuid: "c256f6bc46fd470e9c7aa76162f562a8" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 41193 } }
I20250109 21:08:45.218887 31783 heartbeater.cc:502] Master 127.12.126.126:46267 was elected leader, sending a full tablet report...
I20250109 21:08:45.219113 31933 ts_tablet_manager.cc:1428] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250109 21:08:45.219118 31931 ts_tablet_manager.cc:1428] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250109 21:08:45.219451 31914 heartbeater.cc:502] Master 127.12.126.126:46267 was elected leader, sending a full tablet report...
I20250109 21:08:45.220074 31932 raft_consensus.cc:357] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c17bb55527494eab8d8286d9e91ee4" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 35705 } } peers { permanent_uuid: "11f19c9f345e48478dc46eff27952481" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 36741 } } peers { permanent_uuid: "c256f6bc46fd470e9c7aa76162f562a8" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 41193 } }
I20250109 21:08:45.220266 31932 raft_consensus.cc:383] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:45.220321 31932 raft_consensus.cc:738] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f19c9f345e48478dc46eff27952481, State: Initialized, Role: FOLLOWER
I20250109 21:08:45.220674 31932 consensus_queue.cc:260] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c17bb55527494eab8d8286d9e91ee4" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 35705 } } peers { permanent_uuid: "11f19c9f345e48478dc46eff27952481" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 36741 } } peers { permanent_uuid: "c256f6bc46fd470e9c7aa76162f562a8" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 41193 } }
I20250109 21:08:45.221334 31652 heartbeater.cc:502] Master 127.12.126.126:46267 was elected leader, sending a full tablet report...
I20250109 21:08:45.221427 31932 ts_tablet_manager.cc:1428] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20250109 21:08:45.237232 31784 tablet.cc:2367] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250109 21:08:45.323871 31653 tablet.cc:2367] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250109 21:08:45.413002 31915 tablet.cc:2367] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:08:45.425107 31939 raft_consensus.cc:491] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:08:45.425321 31939 raft_consensus.cc:513] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c17bb55527494eab8d8286d9e91ee4" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 35705 } } peers { permanent_uuid: "11f19c9f345e48478dc46eff27952481" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 36741 } } peers { permanent_uuid: "c256f6bc46fd470e9c7aa76162f562a8" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 41193 } }
I20250109 21:08:45.426491 31939 leader_election.cc:290] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c17bb55527494eab8d8286d9e91ee4 (127.12.126.66:35705), c256f6bc46fd470e9c7aa76162f562a8 (127.12.126.67:41193)
I20250109 21:08:45.430284 31941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:45.426698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:35705 (local address 127.12.126.65:53009)
0109 21:08:45.427167 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:45.427179 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:45.427249 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:45.427473 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:45.427476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:45.427531 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:45.427882 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:45.427890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.428909 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:45.428912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:45.429821 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:45.429831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.429941 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:45.429961 (+ 20us) client_negotiation.cc:769] Sending connection context
0109 21:08:45.430011 (+ 50us) client_negotiation.cc:240] Negotiation successful
0109 21:08:45.430070 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":121,"spinlock_wait_cycles":35456,"thread_start_us":61,"threads_started":1}
I20250109 21:08:45.430286 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:45.426698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:41193 (local address 127.12.126.65:39517)
0109 21:08:45.427078 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:45.427094 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:45.427203 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:45.427473 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:45.427476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:45.427530 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:45.427882 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:45.427890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.428909 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:45.428912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:45.429829 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:45.429834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.429941 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:45.429961 (+ 20us) client_negotiation.cc:769] Sending connection context
0109 21:08:45.430011 (+ 50us) client_negotiation.cc:240] Negotiation successful
0109 21:08:45.430070 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"spinlock_wait_cycles":27136,"thread_start_us":119,"threads_started":1}
I20250109 21:08:45.430586 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:45.426885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:39517 (local address 127.12.126.67:41193)
0109 21:08:45.427010 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:45.427014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:45.427159 (+ 145us) server_negotiation.cc:407] Connection header received
0109 21:08:45.427305 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:45.427310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:45.427363 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:45.427462 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:45.427995 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.428805 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:45.429928 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.430357 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:45.430390 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:45.430437 (+ 47us) server_negotiation.cc:299] Negotiation successful
0109 21:08:45.430482 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20250109 21:08:45.430586 31927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:45.426841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:53009 (local address 127.12.126.66:35705)
0109 21:08:45.426981 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:45.426985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:45.427234 (+ 249us) server_negotiation.cc:407] Connection header received
0109 21:08:45.427332 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:45.427335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:45.427386 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:45.427469 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:45.427995 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.428799 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:45.429929 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.430357 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:45.430390 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:45.430437 (+ 47us) server_negotiation.cc:299] Negotiation successful
0109 21:08:45.430482 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250109 21:08:45.431124 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c602e0fecea4f0abd6d590201aaa264" candidate_uuid: "11f19c9f345e48478dc46eff27952481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c256f6bc46fd470e9c7aa76162f562a8" is_pre_election: true
I20250109 21:08:45.431371 31737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c602e0fecea4f0abd6d590201aaa264" candidate_uuid: "11f19c9f345e48478dc46eff27952481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c17bb55527494eab8d8286d9e91ee4" is_pre_election: true
I20250109 21:08:45.431402 31868 raft_consensus.cc:2463] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11f19c9f345e48478dc46eff27952481 in term 0.
I20250109 21:08:45.431602 31737 raft_consensus.cc:2463] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11f19c9f345e48478dc46eff27952481 in term 0.
I20250109 21:08:45.431833 31539 leader_election.cc:304] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f19c9f345e48478dc46eff27952481, c256f6bc46fd470e9c7aa76162f562a8; no voters:
I20250109 21:08:45.432078 31939 raft_consensus.cc:2798] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:08:45.432144 31939 raft_consensus.cc:491] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:08:45.432201 31939 raft_consensus.cc:3054] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:45.433073 31939 raft_consensus.cc:513] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c17bb55527494eab8d8286d9e91ee4" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 35705 } } peers { permanent_uuid: "11f19c9f345e48478dc46eff27952481" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 36741 } } peers { permanent_uuid: "c256f6bc46fd470e9c7aa76162f562a8" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 41193 } }
I20250109 21:08:45.433390 31939 leader_election.cc:290] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [CANDIDATE]: Term 1 election: Requested vote from peers f3c17bb55527494eab8d8286d9e91ee4 (127.12.126.66:35705), c256f6bc46fd470e9c7aa76162f562a8 (127.12.126.67:41193)
I20250109 21:08:45.433684 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c602e0fecea4f0abd6d590201aaa264" candidate_uuid: "11f19c9f345e48478dc46eff27952481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c256f6bc46fd470e9c7aa76162f562a8"
I20250109 21:08:45.433691 31737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c602e0fecea4f0abd6d590201aaa264" candidate_uuid: "11f19c9f345e48478dc46eff27952481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c17bb55527494eab8d8286d9e91ee4"
I20250109 21:08:45.433779 31868 raft_consensus.cc:3054] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:45.433842 31737 raft_consensus.cc:3054] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:45.434725 31737 raft_consensus.cc:2463] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11f19c9f345e48478dc46eff27952481 in term 1.
I20250109 21:08:45.434767 31868 raft_consensus.cc:2463] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11f19c9f345e48478dc46eff27952481 in term 1.
I20250109 21:08:45.435052 31541 leader_election.cc:304] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f19c9f345e48478dc46eff27952481, f3c17bb55527494eab8d8286d9e91ee4; no voters:
I20250109 21:08:45.435283 31939 raft_consensus.cc:2798] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:45.435478 31939 raft_consensus.cc:695] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [term 1 LEADER]: Becoming Leader. State: Replica: 11f19c9f345e48478dc46eff27952481, State: Running, Role: LEADER
I20250109 21:08:45.435808 31939 consensus_queue.cc:237] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c17bb55527494eab8d8286d9e91ee4" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 35705 } } peers { permanent_uuid: "11f19c9f345e48478dc46eff27952481" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 36741 } } peers { permanent_uuid: "c256f6bc46fd470e9c7aa76162f562a8" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 41193 } }
I20250109 21:08:45.438719 31463 catalog_manager.cc:5526] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11f19c9f345e48478dc46eff27952481 (127.12.126.65). New cstate: current_term: 1 leader_uuid: "11f19c9f345e48478dc46eff27952481" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c17bb55527494eab8d8286d9e91ee4" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 35705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11f19c9f345e48478dc46eff27952481" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 36741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c256f6bc46fd470e9c7aa76162f562a8" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 41193 } health_report { overall_health: UNKNOWN } } }
I20250109 21:08:45.489910 31928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:45.486928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.12.126.65:36741)
0109 21:08:45.487063 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:45.487067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:45.487080 (+ 13us) server_negotiation.cc:407] Connection header received
0109 21:08:45.487177 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:45.487180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:45.487234 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:45.487343 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:08:45.487630 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.488131 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:45.488733 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:45.488871 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:45.489108 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:08:45.489507 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:08:45.489628 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:45.489709 (+ 81us) server_negotiation.cc:299] Negotiation successful
0109 21:08:45.489746 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250109 21:08:45.495285 31737 raft_consensus.cc:1270] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4 [term 1 FOLLOWER]: Refusing update from remote peer 11f19c9f345e48478dc46eff27952481: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:08:45.495285 31868 raft_consensus.cc:1270] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8 [term 1 FOLLOWER]: Refusing update from remote peer 11f19c9f345e48478dc46eff27952481: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:08:45.495906 31939 consensus_queue.cc:1035] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c17bb55527494eab8d8286d9e91ee4" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 35705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:08:45.496083 31942 consensus_queue.cc:1035] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c256f6bc46fd470e9c7aa76162f562a8" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 41193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:08:45.504561 31947 mvcc.cc:204] Tried to move back new op lower bound from 7112527566822072320 to 7112527566587510784. Current Snapshot: MvccSnapshot[applied={T|T < 7112527566822072320 or (T in {7112527566822072320})}]
I20250109 21:08:45.511822 31464 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:08:45.511986 31464 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:08:45.514382 31464 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 67bd68d8d4bb48b0b10bd9fbb5540fa2: Sending DeleteTablet for 3 replicas of tablet 4c602e0fecea4f0abd6d590201aaa264
I20250109 21:08:45.515054 31717 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c602e0fecea4f0abd6d590201aaa264 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:45 UTC) from {username='slave'} at 127.0.0.1:60414
I20250109 21:08:45.515072 31586 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c602e0fecea4f0abd6d590201aaa264 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:45 UTC) from {username='slave'} at 127.0.0.1:34274
I20250109 21:08:45.515496 31955 tablet_replica.cc:331] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481: stopping tablet replica
I20250109 21:08:45.515491 31956 tablet_replica.cc:331] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4: stopping tablet replica
I20250109 21:08:45.515682 31848 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c602e0fecea4f0abd6d590201aaa264 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:45 UTC) from {username='slave'} at 127.0.0.1:32954
I20250109 21:08:45.515736 31956 raft_consensus.cc:2238] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:08:45.515736 31955 raft_consensus.cc:2238] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:08:45.516024 31956 raft_consensus.cc:2267] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:45.516242 31955 raft_consensus.cc:2267] T 4c602e0fecea4f0abd6d590201aaa264 P 11f19c9f345e48478dc46eff27952481 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:45.516587 31957 tablet_replica.cc:331] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8: stopping tablet replica
I20250109 21:08:45.516813 31957 raft_consensus.cc:2238] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:08:45.517077 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 31524
I20250109 21:08:45.517091 31957 raft_consensus.cc:2267] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:45.519424 31957 ts_tablet_manager.cc:1905] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:08:45.521919 31957 ts_tablet_manager.cc:1918] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:08:45.522011 31957 log.cc:1198] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/4c602e0fecea4f0abd6d590201aaa264
I20250109 21:08:45.522341 31957 ts_tablet_manager.cc:1939] T 4c602e0fecea4f0abd6d590201aaa264 P c256f6bc46fd470e9c7aa76162f562a8: Deleting consensus metadata
I20250109 21:08:45.522842 31956 ts_tablet_manager.cc:1905] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:08:45.523502 31448 catalog_manager.cc:4872] TS c256f6bc46fd470e9c7aa76162f562a8 (127.12.126.67:41193): tablet 4c602e0fecea4f0abd6d590201aaa264 (table test-table [id=5aed0ba210d94fbeab20a730fe23536a]) successfully deleted
W20250109 21:08:45.524438 31451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.126.65:36741 (error 108)
I20250109 21:08:45.524960 31956 ts_tablet_manager.cc:1918] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:08:45.525068 31956 log.cc:1198] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/4c602e0fecea4f0abd6d590201aaa264
I20250109 21:08:45.525207 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 31655
I20250109 21:08:45.525389 31956 ts_tablet_manager.cc:1939] T 4c602e0fecea4f0abd6d590201aaa264 P f3c17bb55527494eab8d8286d9e91ee4: Deleting consensus metadata
I20250109 21:08:45.525455 31926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:45.525142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:36741 (local address 127.0.0.1:34298)
0109 21:08:45.525282 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:45.525356 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.65:36741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20250109 21:08:45.525846 31451 catalog_manager.cc:4615] TS 11f19c9f345e48478dc46eff27952481 (127.12.126.65:36741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c602e0fecea4f0abd6d590201aaa264: Network error: Client connection negotiation failed: client connection to 127.12.126.65:36741: connect: Connection refused (error 111)
I20250109 21:08:45.531559 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 31786
I20250109 21:08:45.531986 31926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:45.531656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:35705 (local address 127.0.0.1:60428)
0109 21:08:45.531854 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:45.531901 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.66:35705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20250109 21:08:45.537097 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 31433
2025-01-09T21:08:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1120 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:08:45 dist-test-slave-g0zb krb5kdc[31961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:08:45 dist-test-slave-g0zb krb5kdc[31961](info): set up 2 sockets
Jan 09 21:08:45 dist-test-slave-g0zb krb5kdc[31961](info): commencing operation
krb5kdc: starting...
W20250109 21:08:46.519381 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:08:46 dist-test-slave-g0zb krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:08:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:08:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:08:46.623607 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:42219
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:42219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:46.725920 31974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:46.726161 31974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:46.726212 31974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:46.729279 31974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:08:46.729336 31974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:46.729358 31974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:46.729377 31974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:08:46.729393 31974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:08:46.733484 31974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:42219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:42219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:46.734449 31974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:46.735307 31974 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:46.742215 31980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:46.742899 31982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:46.743728 31974 server_base.cc:1034] running on GCE node
W20250109 21:08:46.744199 31979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:46.744737 31974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:46.745626 31974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:46.746789 31974 hybrid_clock.cc:648] HybridClock initialized: now 1736456926746778 us; error 26 us; skew 500 ppm
Jan 09 21:08:46 dist-test-slave-g0zb krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456926, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:46.749447 31974 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:08:46.750278 31974 webserver.cc:458] Webserver started at http://127.12.126.126:33275/ using document root <none> and password file <none>
I20250109 21:08:46.750784 31974 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:46.750870 31974 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:46.751029 31974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:46.752722 31974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "2dc6fd7dc156469592014695dc4b87ba"
format_stamp: "Formatted at 2025-01-09 21:08:46 on dist-test-slave-g0zb"
server_key: "a5bf9edb3213a03badfd820299680747"
server_key_iv: "3d61195c828c33096e9e80b7bee2cb38"
server_key_version: "encryptionkey@0"
I20250109 21:08:46.753163 31974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "2dc6fd7dc156469592014695dc4b87ba"
format_stamp: "Formatted at 2025-01-09 21:08:46 on dist-test-slave-g0zb"
server_key: "a5bf9edb3213a03badfd820299680747"
server_key_iv: "3d61195c828c33096e9e80b7bee2cb38"
server_key_version: "encryptionkey@0"
I20250109 21:08:46.756374 31974 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:08:46.758380 31989 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:46.759359 31974 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:46.759457 31974 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "2dc6fd7dc156469592014695dc4b87ba"
format_stamp: "Formatted at 2025-01-09 21:08:46 on dist-test-slave-g0zb"
server_key: "a5bf9edb3213a03badfd820299680747"
server_key_iv: "3d61195c828c33096e9e80b7bee2cb38"
server_key_version: "encryptionkey@0"
I20250109 21:08:46.759546 31974 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:46.775096 31974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:46.777624 31974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:46.777786 31974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:46.783875 31974 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:42219
I20250109 21:08:46.784109 32041 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:42219 every 8 connection(s)
I20250109 21:08:46.784807 31974 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:08:46.787163 32042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:46.788280 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 31974
I20250109 21:08:46.788427 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:08:46.788698 12793 external_mini_cluster.cc:1453] Setting key 8f95b4f118398a1187d7a828b3422d6d
I20250109 21:08:46.793243 32042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba: Bootstrap starting.
I20250109 21:08:46.795475 32042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:46.796243 32042 log.cc:826] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba: Log is configured to *not* fsync() on all Append() calls
Jan 09 21:08:46 dist-test-slave-g0zb krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:08:46.798105 32042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba: No bootstrap required, opened a new log
I20250109 21:08:46.800908 32042 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc6fd7dc156469592014695dc4b87ba" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42219 } }
I20250109 21:08:46.801086 32042 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:46.801129 32042 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dc6fd7dc156469592014695dc4b87ba, State: Initialized, Role: FOLLOWER
I20250109 21:08:46.801249 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:46.789895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38120 (local address 127.12.126.126:42219)
0109 21:08:46.790290 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:46.790301 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:46.790339 (+ 38us) server_negotiation.cc:407] Connection header received
0109 21:08:46.791068 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:46.791093 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:46.791447 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:46.791835 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:46.792616 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:46.793752 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:46.794397 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:46.794646 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:46.796976 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:46.796992 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:46.797004 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:46.797036 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:46.798983 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:46.799365 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:46.799370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:46.799374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:46.799438 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:46.799644 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:46.799647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:46.799649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:46.799962 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:46.800144 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:46.800403 (+ 259us) server_negotiation.cc:299] Negotiation successful
0109 21:08:46.800631 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":101,"threads_started":1}
I20250109 21:08:46.801533 32042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc6fd7dc156469592014695dc4b87ba" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42219 } }
I20250109 21:08:46.801648 32042 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:08:46.801725 32042 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:08:46.801851 32042 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:46.802675 32042 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc6fd7dc156469592014695dc4b87ba" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42219 } }
I20250109 21:08:46.803030 32042 leader_election.cc:304] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dc6fd7dc156469592014695dc4b87ba; no voters:
I20250109 21:08:46.803318 32042 leader_election.cc:290] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:08:46.803433 32047 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:46.803658 32047 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba [term 1 LEADER]: Becoming Leader. State: Replica: 2dc6fd7dc156469592014695dc4b87ba, State: Running, Role: LEADER
I20250109 21:08:46.803927 32047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc6fd7dc156469592014695dc4b87ba" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42219 } }
I20250109 21:08:46.804363 32042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:08:46.805420 32048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dc6fd7dc156469592014695dc4b87ba. Latest consensus state: current_term: 1 leader_uuid: "2dc6fd7dc156469592014695dc4b87ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc6fd7dc156469592014695dc4b87ba" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42219 } } }
I20250109 21:08:46.805420 32049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dc6fd7dc156469592014695dc4b87ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc6fd7dc156469592014695dc4b87ba" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42219 } } }
I20250109 21:08:46.805563 32048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:46.805563 32049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:46.806049 32056 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:08:46.808583 32056 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:08:46.813371 32056 catalog_manager.cc:1348] Generated new cluster ID: 8e1edad64d8d440a9bf4fc7cb1f44ffe
I20250109 21:08:46.813432 32056 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:08:46.829689 32056 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:08:46.830869 32056 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:08:46.843607 32056 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba: Generated new TSK 0
I20250109 21:08:46.844367 32056 catalog_manager.cc:1515] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.126.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
Entry for principal kudu/127.12.126.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab.
I20250109 21:08:46.869824 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42219
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:46.975028 32068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:46.975240 32068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:46.975323 32068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:46.978389 32068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:46.978448 32068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:46.978551 32068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:08:46.982650 32068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.65.keytab
--principal=kudu/127.12.126.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:46.983682 32068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:46.984544 32068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:46.992906 32074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:46.993069 32075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:46.993643 32077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:46.994477 32068 server_base.cc:1034] running on GCE node
I20250109 21:08:46.994855 32068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:46.995450 32068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:46.996590 32068 hybrid_clock.cc:648] HybridClock initialized: now 1736456926996572 us; error 38 us; skew 500 ppm
Jan 09 21:08:46 dist-test-slave-g0zb krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456926, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:46.999157 32068 init.cc:377] Logged in from keytab as kudu/127.12.126.65@KRBTEST.COM (short username kudu)
I20250109 21:08:46.999964 32068 webserver.cc:458] Webserver started at http://127.12.126.65:35153/ using document root <none> and password file <none>
I20250109 21:08:47.000468 32068 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:47.000516 32068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:47.000666 32068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:47.002430 32068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "437cbabc1ed5447a9a555c1b666c922b"
format_stamp: "Formatted at 2025-01-09 21:08:46 on dist-test-slave-g0zb"
server_key: "79cc73400655620829b4ec16b0142461"
server_key_iv: "590dde4b86a5797bc1f08f5a45419145"
server_key_version: "encryptionkey@0"
I20250109 21:08:47.002916 32068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "437cbabc1ed5447a9a555c1b666c922b"
format_stamp: "Formatted at 2025-01-09 21:08:46 on dist-test-slave-g0zb"
server_key: "79cc73400655620829b4ec16b0142461"
server_key_iv: "590dde4b86a5797bc1f08f5a45419145"
server_key_version: "encryptionkey@0"
I20250109 21:08:47.006125 32068 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250109 21:08:47.008227 32084 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:47.009176 32068 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:08:47.009280 32068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "437cbabc1ed5447a9a555c1b666c922b"
format_stamp: "Formatted at 2025-01-09 21:08:46 on dist-test-slave-g0zb"
server_key: "79cc73400655620829b4ec16b0142461"
server_key_iv: "590dde4b86a5797bc1f08f5a45419145"
server_key_version: "encryptionkey@0"
I20250109 21:08:47.009380 32068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:47.024632 32068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:47.027937 32068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:47.028110 32068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:47.028601 32068 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:47.029371 32068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:47.029423 32068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:47.029484 32068 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:47.029524 32068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:47.037356 32068 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:46353
I20250109 21:08:47.037408 32197 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:46353 every 8 connection(s)
I20250109 21:08:47.038281 32068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:08:47.045195 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 32068
I20250109 21:08:47.045331 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:08:47.045624 12793 external_mini_cluster.cc:1453] Setting key 53e6596a2c7f4822039ec63c9a3e0e4b
Jan 09 21:08:47 dist-test-slave-g0zb krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456926, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.65@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:08:47.051148 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.039978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:50871 (local address 127.12.126.126:42219)
0109 21:08:47.040129 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:47.040132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:47.040740 (+ 608us) server_negotiation.cc:407] Connection header received
0109 21:08:47.041992 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:47.041995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:47.042065 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:47.042142 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:47.043407 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.043898 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.044523 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.044701 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.047289 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:47.047306 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:47.047309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:47.047337 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:47.049064 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:47.049658 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:47.049660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:47.049662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:47.049699 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:47.049961 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:47.049966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:47.049971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:47.050174 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:47.050295 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:47.050737 (+ 442us) server_negotiation.cc:299] Negotiation successful
0109 21:08:47.050910 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20250109 21:08:47.051671 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.040226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42219 (local address 127.12.126.65:50871)
0109 21:08:47.040613 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:47.040646 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:47.041780 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:47.042305 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:47.042312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:47.042672 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:47.043262 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:47.043275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.044017 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.044020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:47.044416 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:47.044422 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.044620 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.045230 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:08:47.045266 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:47.047138 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:47.049195 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:47.049202 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:47.049217 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:47.049560 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:47.049781 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:47.049783 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:47.049785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:47.049877 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:47.050286 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:47.050291 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:47.050511 (+ 220us) client_negotiation.cc:769] Sending connection context
0109 21:08:47.050669 (+ 158us) client_negotiation.cc:240] Negotiation successful
0109 21:08:47.050954 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":91,"threads_started":1}
I20250109 21:08:47.052798 32198 heartbeater.cc:346] Connected to a master server at 127.12.126.126:42219
I20250109 21:08:47.053046 32198 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:47.053653 32198 heartbeater.cc:510] Master 127.12.126.126:42219 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.126.66@KRBTEST.COM; defaulting to no policy
I20250109 21:08:47.055366 32006 ts_manager.cc:194] Registered new tserver with Master: 437cbabc1ed5447a9a555c1b666c922b (127.12.126.65:46353)
I20250109 21:08:47.056551 32006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.65@KRBTEST.COM'} at 127.12.126.65:50871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
Entry for principal kudu/127.12.126.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab.
I20250109 21:08:47.074355 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42219
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:47.177273 32203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:47.177501 32203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:47.177592 32203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:47.180701 32203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:47.180771 32203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:47.180886 32203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:08:47.185083 32203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.66.keytab
--principal=kudu/127.12.126.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:47.186120 32203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:47.186967 32203 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:47.195446 32211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:47.195569 32208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:47.196292 32209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:47.197036 32203 server_base.cc:1034] running on GCE node
I20250109 21:08:47.197389 32203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:47.197945 32203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:47.199100 32203 hybrid_clock.cc:648] HybridClock initialized: now 1736456927199087 us; error 31 us; skew 500 ppm
Jan 09 21:08:47 dist-test-slave-g0zb krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456927, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:47.201881 32203 init.cc:377] Logged in from keytab as kudu/127.12.126.66@KRBTEST.COM (short username kudu)
I20250109 21:08:47.202731 32203 webserver.cc:458] Webserver started at http://127.12.126.66:33829/ using document root <none> and password file <none>
I20250109 21:08:47.203301 32203 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:47.203354 32203 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:47.203505 32203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:47.205233 32203 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "cb0aef9c42b54f00a100de5e833e85e9"
format_stamp: "Formatted at 2025-01-09 21:08:47 on dist-test-slave-g0zb"
server_key: "c4e5918fbd94b070991003d20dfb418c"
server_key_iv: "4059fcc7fd77b1c204d8f44ea3bf8c15"
server_key_version: "encryptionkey@0"
I20250109 21:08:47.205720 32203 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "cb0aef9c42b54f00a100de5e833e85e9"
format_stamp: "Formatted at 2025-01-09 21:08:47 on dist-test-slave-g0zb"
server_key: "c4e5918fbd94b070991003d20dfb418c"
server_key_iv: "4059fcc7fd77b1c204d8f44ea3bf8c15"
server_key_version: "encryptionkey@0"
I20250109 21:08:47.209048 32203 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250109 21:08:47.211059 32218 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:47.212024 32203 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 21:08:47.212112 32203 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "cb0aef9c42b54f00a100de5e833e85e9"
format_stamp: "Formatted at 2025-01-09 21:08:47 on dist-test-slave-g0zb"
server_key: "c4e5918fbd94b070991003d20dfb418c"
server_key_iv: "4059fcc7fd77b1c204d8f44ea3bf8c15"
server_key_version: "encryptionkey@0"
I20250109 21:08:47.212206 32203 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:47.226882 32203 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:47.229391 32203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:47.229552 32203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:47.230052 32203 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:47.230834 32203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:47.230881 32203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:47.230923 32203 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:47.230957 32203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:47.238973 32203 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:41431
I20250109 21:08:47.239005 32331 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:41431 every 8 connection(s)
I20250109 21:08:47.239876 32203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
Jan 09 21:08:47 dist-test-slave-g0zb krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456927, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.66@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:08:47.249410 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 32203
I20250109 21:08:47.249531 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:08:47.249786 12793 external_mini_cluster.cc:1453] Setting key eecfbba597be9a5ab33a29f827d16ba6
I20250109 21:08:47.251145 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.241483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:50229 (local address 127.12.126.126:42219)
0109 21:08:47.241624 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:47.241628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:47.242290 (+ 662us) server_negotiation.cc:407] Connection header received
0109 21:08:47.243484 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:47.243488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:47.243538 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:47.243612 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:47.244748 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.245237 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.245797 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.245918 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.247959 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:47.247977 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:47.247980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:47.248004 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:47.249359 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:47.249864 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:47.249868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:47.249870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:47.249914 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:47.250166 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:47.250170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:47.250172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:47.250310 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:47.250397 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:47.250864 (+ 467us) server_negotiation.cc:299] Negotiation successful
0109 21:08:47.250978 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250109 21:08:47.251708 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.241752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42219 (local address 127.12.126.66:50229)
0109 21:08:47.242153 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:47.242186 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:47.243297 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:47.243757 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:47.243764 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:47.244111 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:47.244634 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:47.244647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.245326 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.245329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:47.245711 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:47.245717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.245880 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.246394 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:08:47.246414 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:47.247817 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:47.249473 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:47.249478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:47.249487 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:47.249758 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:47.249993 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:47.249998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:47.250000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:47.250080 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:47.250406 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:47.250414 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:47.250623 (+ 209us) client_negotiation.cc:769] Sending connection context
0109 21:08:47.250819 (+ 196us) client_negotiation.cc:240] Negotiation successful
0109 21:08:47.251073 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":94,"threads_started":1}
I20250109 21:08:47.252679 32332 heartbeater.cc:346] Connected to a master server at 127.12.126.126:42219
I20250109 21:08:47.252885 32332 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:47.253342 32332 heartbeater.cc:510] Master 127.12.126.126:42219 requested a full tablet report, sending...
I20250109 21:08:47.254287 32006 ts_manager.cc:194] Registered new tserver with Master: cb0aef9c42b54f00a100de5e833e85e9 (127.12.126.66:41431)
I20250109 21:08:47.255101 32006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.66@KRBTEST.COM'} at 127.12.126.66:50229
WARNING: no policy specified for kudu/127.12.126.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
Entry for principal kudu/127.12.126.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab.
I20250109 21:08:47.273815 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42219
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:47.377691 32337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:47.377938 32337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:47.378026 32337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:47.381115 32337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:47.381183 32337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:47.381287 32337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:08:47.385406 32337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.67.keytab
--principal=kudu/127.12.126.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:42219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:47.386430 32337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:47.387277 32337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:47.394901 32343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:47.395545 32342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:47.396344 32345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:47.397135 32337 server_base.cc:1034] running on GCE node
I20250109 21:08:47.397435 32337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:47.397913 32337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:47.399053 32337 hybrid_clock.cc:648] HybridClock initialized: now 1736456927399029 us; error 40 us; skew 500 ppm
Jan 09 21:08:47 dist-test-slave-g0zb krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456927, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:47.401501 32337 init.cc:377] Logged in from keytab as kudu/127.12.126.67@KRBTEST.COM (short username kudu)
I20250109 21:08:47.402341 32337 webserver.cc:458] Webserver started at http://127.12.126.67:41467/ using document root <none> and password file <none>
I20250109 21:08:47.402875 32337 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:47.402931 32337 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:47.403087 32337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:47.404774 32337 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "1fa346abef3745939a9bacbcac7898d9"
format_stamp: "Formatted at 2025-01-09 21:08:47 on dist-test-slave-g0zb"
server_key: "1613494874007938e81239680e44711c"
server_key_iv: "17d2d17880970e4cfbd1747426dba76b"
server_key_version: "encryptionkey@0"
I20250109 21:08:47.405195 32337 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "1fa346abef3745939a9bacbcac7898d9"
format_stamp: "Formatted at 2025-01-09 21:08:47 on dist-test-slave-g0zb"
server_key: "1613494874007938e81239680e44711c"
server_key_iv: "17d2d17880970e4cfbd1747426dba76b"
server_key_version: "encryptionkey@0"
I20250109 21:08:47.408381 32337 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:08:47.410378 32352 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:47.411353 32337 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:47.411445 32337 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "1fa346abef3745939a9bacbcac7898d9"
format_stamp: "Formatted at 2025-01-09 21:08:47 on dist-test-slave-g0zb"
server_key: "1613494874007938e81239680e44711c"
server_key_iv: "17d2d17880970e4cfbd1747426dba76b"
server_key_version: "encryptionkey@0"
I20250109 21:08:47.411559 32337 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:47.424919 32337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:47.427417 32337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:47.427570 32337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:47.428056 32337 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:47.428783 32337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:47.428828 32337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:47.428882 32337 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:47.428915 32337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:47.437099 32337 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:43503
I20250109 21:08:47.437124 32465 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:43503 every 8 connection(s)
I20250109 21:08:47.437978 32337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:08:47.439069 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 32337
I20250109 21:08:47.439188 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:08:47.439438 12793 external_mini_cluster.cc:1453] Setting key 3c3963625e2a5312c2381342246e5b36
Jan 09 21:08:47 dist-test-slave-g0zb krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456927, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.67@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:08:47.449729 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.439737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:55253 (local address 127.12.126.126:42219)
0109 21:08:47.439884 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:47.439891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:47.440664 (+ 773us) server_negotiation.cc:407] Connection header received
0109 21:08:47.442086 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:47.442090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:47.442144 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:47.442224 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:47.443459 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.443981 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.444573 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.444748 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.446737 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:47.446751 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:47.446753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:47.446776 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:47.448043 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:47.448486 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:47.448491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:47.448493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:47.448534 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:47.448781 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:47.448790 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:47.448794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:47.448940 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:47.449024 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:47.449464 (+ 440us) server_negotiation.cc:299] Negotiation successful
0109 21:08:47.449574 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250109 21:08:47.450312 32468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.440068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42219 (local address 127.12.126.67:55253)
0109 21:08:47.440536 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:47.440581 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:47.441895 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:47.442369 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:47.442377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:47.442730 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:47.443330 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:47.443342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.444093 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.444096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:47.444469 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:47.444476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.444674 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.445253 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:08:47.445276 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:47.446595 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:47.448150 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:47.448155 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:47.448165 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:47.448392 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:47.448613 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:08:47.448618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:08:47.448620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:08:47.448697 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:08:47.449041 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:47.449047 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:47.449248 (+ 201us) client_negotiation.cc:769] Sending connection context
0109 21:08:47.449413 (+ 165us) client_negotiation.cc:240] Negotiation successful
0109 21:08:47.449655 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":127,"threads_started":1}
I20250109 21:08:47.451236 32466 heartbeater.cc:346] Connected to a master server at 127.12.126.126:42219
I20250109 21:08:47.451448 32466 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:47.451969 32466 heartbeater.cc:510] Master 127.12.126.126:42219 requested a full tablet report, sending...
I20250109 21:08:47.452929 32006 ts_manager.cc:194] Registered new tserver with Master: 1fa346abef3745939a9bacbcac7898d9 (127.12.126.67:43503)
I20250109 21:08:47.453482 32006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.126.67@KRBTEST.COM'} at 127.12.126.67:55253
I20250109 21:08:47.464710 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 21:08:47.472702 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.466464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.12.126.126:42219)
0109 21:08:47.466621 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:47.466625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:47.466743 (+ 118us) server_negotiation.cc:407] Connection header received
0109 21:08:47.466903 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:47.466908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:47.466950 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:47.467013 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:47.467822 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.468291 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.468902 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.469096 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.470179 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:47.470192 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:47.470194 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:47.470215 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:47.471352 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:47.471700 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:47.471704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:47.471705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:47.471752 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:47.471945 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:47.471948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:47.471950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:47.472103 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:47.472219 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:47.472421 (+ 202us) server_negotiation.cc:299] Negotiation successful
0109 21:08:47.472536 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20250109 21:08:47.475281 32006 catalog_manager.cc:1909] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:08:47.477754 32006 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:08:47.489918 32478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.485617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:41431 (local address 127.0.0.1:45532)
0109 21:08:47.486249 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:47.486288 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:47.486416 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:47.487287 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:47.487290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:47.487309 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:47.487557 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:47.487565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.488561 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.488564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:47.489397 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:47.489406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.489510 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.489567 (+ 57us) client_negotiation.cc:769] Sending connection context
0109 21:08:47.489676 (+ 109us) client_negotiation.cc:240] Negotiation successful
0109 21:08:47.489776 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":107,"threads_started":1}
I20250109 21:08:47.490198 32477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.485819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:46353 (local address 127.0.0.1:56174)
0109 21:08:47.486651 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:47.486663 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:47.486728 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:47.487482 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:47.487485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:47.487500 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:47.487718 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:47.487725 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.489033 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.489036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:47.489857 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:47.489866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.489944 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.489960 (+ 16us) client_negotiation.cc:769] Sending connection context
0109 21:08:47.490001 (+ 41us) client_negotiation.cc:240] Negotiation successful
0109 21:08:47.490062 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":84,"threads_started":1}
I20250109 21:08:47.490643 32481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.486225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.12.126.66:41431)
0109 21:08:47.486940 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:47.486945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:47.486964 (+ 19us) server_negotiation.cc:407] Connection header received
0109 21:08:47.487032 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:47.487036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:47.487156 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:47.487287 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:47.487709 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.488450 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.489532 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.490076 (+ 544us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.490202 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:47.490407 (+ 205us) server_negotiation.cc:299] Negotiation successful
0109 21:08:47.490494 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":113,"threads_started":1}
I20250109 21:08:47.491247 32480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.485420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:43503 (local address 127.0.0.1:56530)
0109 21:08:47.486249 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:47.486288 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:47.486421 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:47.487226 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:47.487232 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:47.487253 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:47.487554 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:47.487563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.488894 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.488897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:47.489790 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:47.489799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.490864 (+ 1065us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.490880 (+ 16us) client_negotiation.cc:769] Sending connection context
0109 21:08:47.491050 (+ 170us) client_negotiation.cc:240] Negotiation successful
0109 21:08:47.491124 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":72,"threads_started":1}
I20250109 21:08:47.491466 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.485628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.12.126.67:43503)
0109 21:08:47.486003 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:47.486010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:47.486367 (+ 357us) server_negotiation.cc:407] Connection header received
0109 21:08:47.486606 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:47.486630 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:47.486893 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:47.487029 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:47.487679 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.488766 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.489951 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.490673 (+ 722us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.490783 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:47.491066 (+ 283us) server_negotiation.cc:299] Negotiation successful
0109 21:08:47.491163 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":107,"threads_started":1}
I20250109 21:08:47.491524 32482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.486560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56174 (local address 127.12.126.65:46353)
0109 21:08:47.487015 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:47.487021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:47.487043 (+ 22us) server_negotiation.cc:407] Connection header received
0109 21:08:47.487128 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:47.487134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:47.487304 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:47.487481 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:47.487850 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.488922 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.490433 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.491030 (+ 597us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.491178 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:47.491250 (+ 72us) server_negotiation.cc:299] Negotiation successful
0109 21:08:47.491345 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":129,"threads_started":1}
I20250109 21:08:47.493602 32266 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5cdb16277648b3921d739b20bed535 (DEFAULT_TABLE table=test-table [id=12a5f996db894a9aa10eb9bc02be858e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:47.493845 32132 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5cdb16277648b3921d739b20bed535 (DEFAULT_TABLE table=test-table [id=12a5f996db894a9aa10eb9bc02be858e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:47.494650 32132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a5cdb16277648b3921d739b20bed535. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:47.494650 32266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a5cdb16277648b3921d739b20bed535. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:47.496330 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5cdb16277648b3921d739b20bed535 (DEFAULT_TABLE table=test-table [id=12a5f996db894a9aa10eb9bc02be858e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:47.497148 32400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a5cdb16277648b3921d739b20bed535. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:47.501941 32483 tablet_bootstrap.cc:492] T 0a5cdb16277648b3921d739b20bed535 P 437cbabc1ed5447a9a555c1b666c922b: Bootstrap starting.
I20250109 21:08:47.502343 32485 tablet_bootstrap.cc:492] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9: Bootstrap starting.
I20250109 21:08:47.503933 32483 tablet_bootstrap.cc:654] T 0a5cdb16277648b3921d739b20bed535 P 437cbabc1ed5447a9a555c1b666c922b: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:47.504061 32485 tablet_bootstrap.cc:654] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:47.504863 32483 log.cc:826] T 0a5cdb16277648b3921d739b20bed535 P 437cbabc1ed5447a9a555c1b666c922b: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:47.505178 32485 log.cc:826] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:47.505450 32484 tablet_bootstrap.cc:492] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9: Bootstrap starting.
I20250109 21:08:47.507093 32485 tablet_bootstrap.cc:492] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9: No bootstrap required, opened a new log
I20250109 21:08:47.507273 32485 ts_tablet_manager.cc:1397] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:08:47.507462 32484 tablet_bootstrap.cc:654] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:47.507764 32483 tablet_bootstrap.cc:492] T 0a5cdb16277648b3921d739b20bed535 P 437cbabc1ed5447a9a555c1b666c922b: No bootstrap required, opened a new log
I20250109 21:08:47.507933 32483 ts_tablet_manager.cc:1397] T 0a5cdb16277648b3921d739b20bed535 P 437cbabc1ed5447a9a555c1b666c922b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20250109 21:08:47.508436 32484 log.cc:826] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:47.509829 32484 tablet_bootstrap.cc:492] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9: No bootstrap required, opened a new log
I20250109 21:08:47.509980 32484 ts_tablet_manager.cc:1397] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:08:47.510555 32485 raft_consensus.cc:357] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa346abef3745939a9bacbcac7898d9" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43503 } } peers { permanent_uuid: "cb0aef9c42b54f00a100de5e833e85e9" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 41431 } } peers { permanent_uuid: "437cbabc1ed5447a9a555c1b666c922b" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 46353 } }
I20250109 21:08:47.510771 32485 raft_consensus.cc:383] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:47.510850 32485 raft_consensus.cc:738] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa346abef3745939a9bacbcac7898d9, State: Initialized, Role: FOLLOWER
I20250109 21:08:47.511067 32483 raft_consensus.cc:357] T 0a5cdb16277648b3921d739b20bed535 P 437cbabc1ed5447a9a555c1b666c922b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa346abef3745939a9bacbcac7898d9" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43503 } } peers { permanent_uuid: "cb0aef9c42b54f00a100de5e833e85e9" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 41431 } } peers { permanent_uuid: "437cbabc1ed5447a9a555c1b666c922b" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 46353 } }
I20250109 21:08:47.511253 32483 raft_consensus.cc:383] T 0a5cdb16277648b3921d739b20bed535 P 437cbabc1ed5447a9a555c1b666c922b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:47.511302 32483 raft_consensus.cc:738] T 0a5cdb16277648b3921d739b20bed535 P 437cbabc1ed5447a9a555c1b666c922b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437cbabc1ed5447a9a555c1b666c922b, State: Initialized, Role: FOLLOWER
I20250109 21:08:47.511298 32485 consensus_queue.cc:260] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa346abef3745939a9bacbcac7898d9" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43503 } } peers { permanent_uuid: "cb0aef9c42b54f00a100de5e833e85e9" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 41431 } } peers { permanent_uuid: "437cbabc1ed5447a9a555c1b666c922b" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 46353 } }
I20250109 21:08:47.511755 32483 consensus_queue.cc:260] T 0a5cdb16277648b3921d739b20bed535 P 437cbabc1ed5447a9a555c1b666c922b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa346abef3745939a9bacbcac7898d9" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43503 } } peers { permanent_uuid: "cb0aef9c42b54f00a100de5e833e85e9" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 41431 } } peers { permanent_uuid: "437cbabc1ed5447a9a555c1b666c922b" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 46353 } }
I20250109 21:08:47.511986 32466 heartbeater.cc:502] Master 127.12.126.126:42219 was elected leader, sending a full tablet report...
I20250109 21:08:47.512179 32485 ts_tablet_manager.cc:1428] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:08:47.512485 32483 ts_tablet_manager.cc:1428] T 0a5cdb16277648b3921d739b20bed535 P 437cbabc1ed5447a9a555c1b666c922b: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20250109 21:08:47.512812 32198 heartbeater.cc:502] Master 127.12.126.126:42219 was elected leader, sending a full tablet report...
I20250109 21:08:47.512828 32484 raft_consensus.cc:357] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa346abef3745939a9bacbcac7898d9" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43503 } } peers { permanent_uuid: "cb0aef9c42b54f00a100de5e833e85e9" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 41431 } } peers { permanent_uuid: "437cbabc1ed5447a9a555c1b666c922b" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 46353 } }
I20250109 21:08:47.513005 32484 raft_consensus.cc:383] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:47.513060 32484 raft_consensus.cc:738] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0aef9c42b54f00a100de5e833e85e9, State: Initialized, Role: FOLLOWER
I20250109 21:08:47.513408 32484 consensus_queue.cc:260] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa346abef3745939a9bacbcac7898d9" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43503 } } peers { permanent_uuid: "cb0aef9c42b54f00a100de5e833e85e9" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 41431 } } peers { permanent_uuid: "437cbabc1ed5447a9a555c1b666c922b" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 46353 } }
I20250109 21:08:47.514215 32484 ts_tablet_manager.cc:1428] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250109 21:08:47.514554 32332 heartbeater.cc:502] Master 127.12.126.126:42219 was elected leader, sending a full tablet report...
I20250109 21:08:47.534674 32491 raft_consensus.cc:491] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:08:47.534909 32491 raft_consensus.cc:513] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa346abef3745939a9bacbcac7898d9" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43503 } } peers { permanent_uuid: "cb0aef9c42b54f00a100de5e833e85e9" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 41431 } } peers { permanent_uuid: "437cbabc1ed5447a9a555c1b666c922b" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 46353 } }
I20250109 21:08:47.536023 32491 leader_election.cc:290] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fa346abef3745939a9bacbcac7898d9 (127.12.126.67:43503), 437cbabc1ed5447a9a555c1b666c922b (127.12.126.65:46353)
I20250109 21:08:47.538869 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.536222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:43503 (local address 127.12.126.66:37705)
0109 21:08:47.536345 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:47.536362 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:47.536448 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:47.536723 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:47.536726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:47.536776 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:47.537033 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:47.537042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.537874 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.537878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:47.538519 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:47.538528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.538620 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.538637 (+ 17us) client_negotiation.cc:769] Sending connection context
0109 21:08:47.538674 (+ 37us) client_negotiation.cc:240] Negotiation successful
0109 21:08:47.538715 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"spinlock_wait_cycles":34048}
I20250109 21:08:47.539175 32492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.536262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:46353 (local address 127.12.126.66:32931)
0109 21:08:47.536650 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:47.536666 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:47.536786 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:47.537035 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:47.537037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:47.537050 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:47.537356 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:47.537364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.538115 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.538117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:47.538771 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:47.538780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.538904 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.538922 (+ 18us) client_negotiation.cc:769] Sending connection context
0109 21:08:47.538971 (+ 49us) client_negotiation.cc:240] Negotiation successful
0109 21:08:47.539033 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"spinlock_wait_cycles":10240,"thread_start_us":114,"threads_started":1}
I20250109 21:08:47.539412 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.536324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:37705 (local address 127.12.126.67:43503)
0109 21:08:47.536475 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:47.536479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:47.536495 (+ 16us) server_negotiation.cc:407] Connection header received
0109 21:08:47.536555 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:47.536560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:47.536608 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:47.536699 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:47.537153 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.537772 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.538634 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.539131 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.539171 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:47.539224 (+ 53us) server_negotiation.cc:299] Negotiation successful
0109 21:08:47.539291 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20250109 21:08:47.539495 32199 tablet.cc:2367] T 0a5cdb16277648b3921d739b20bed535 P 437cbabc1ed5447a9a555c1b666c922b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:08:47.539633 32482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.536504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:32931 (local address 127.12.126.65:46353)
0109 21:08:47.536618 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:47.536621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:47.536729 (+ 108us) server_negotiation.cc:407] Connection header received
0109 21:08:47.536890 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:47.536894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:47.536943 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:47.537020 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:47.537463 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.538002 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.538897 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.539380 (+ 483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.539416 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:47.539477 (+ 61us) server_negotiation.cc:299] Negotiation successful
0109 21:08:47.539537 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250109 21:08:47.539986 32420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5cdb16277648b3921d739b20bed535" candidate_uuid: "cb0aef9c42b54f00a100de5e833e85e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa346abef3745939a9bacbcac7898d9" is_pre_election: true
I20250109 21:08:47.540096 32152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5cdb16277648b3921d739b20bed535" candidate_uuid: "cb0aef9c42b54f00a100de5e833e85e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437cbabc1ed5447a9a555c1b666c922b" is_pre_election: true
I20250109 21:08:47.540246 32420 raft_consensus.cc:2463] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb0aef9c42b54f00a100de5e833e85e9 in term 0.
I20250109 21:08:47.540344 32152 raft_consensus.cc:2463] T 0a5cdb16277648b3921d739b20bed535 P 437cbabc1ed5447a9a555c1b666c922b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb0aef9c42b54f00a100de5e833e85e9 in term 0.
I20250109 21:08:47.540710 32219 leader_election.cc:304] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa346abef3745939a9bacbcac7898d9, cb0aef9c42b54f00a100de5e833e85e9; no voters:
I20250109 21:08:47.540949 32491 raft_consensus.cc:2798] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:08:47.541020 32491 raft_consensus.cc:491] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:08:47.541069 32491 raft_consensus.cc:3054] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:47.542044 32491 raft_consensus.cc:513] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa346abef3745939a9bacbcac7898d9" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43503 } } peers { permanent_uuid: "cb0aef9c42b54f00a100de5e833e85e9" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 41431 } } peers { permanent_uuid: "437cbabc1ed5447a9a555c1b666c922b" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 46353 } }
I20250109 21:08:47.542362 32491 leader_election.cc:290] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [CANDIDATE]: Term 1 election: Requested vote from peers 1fa346abef3745939a9bacbcac7898d9 (127.12.126.67:43503), 437cbabc1ed5447a9a555c1b666c922b (127.12.126.65:46353)
I20250109 21:08:47.542686 32152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5cdb16277648b3921d739b20bed535" candidate_uuid: "cb0aef9c42b54f00a100de5e833e85e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437cbabc1ed5447a9a555c1b666c922b"
I20250109 21:08:47.542693 32420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5cdb16277648b3921d739b20bed535" candidate_uuid: "cb0aef9c42b54f00a100de5e833e85e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa346abef3745939a9bacbcac7898d9"
I20250109 21:08:47.542784 32152 raft_consensus.cc:3054] T 0a5cdb16277648b3921d739b20bed535 P 437cbabc1ed5447a9a555c1b666c922b [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:47.542820 32420 raft_consensus.cc:3054] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:47.543699 32152 raft_consensus.cc:2463] T 0a5cdb16277648b3921d739b20bed535 P 437cbabc1ed5447a9a555c1b666c922b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0aef9c42b54f00a100de5e833e85e9 in term 1.
I20250109 21:08:47.543738 32420 raft_consensus.cc:2463] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0aef9c42b54f00a100de5e833e85e9 in term 1.
I20250109 21:08:47.543983 32221 leader_election.cc:304] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 437cbabc1ed5447a9a555c1b666c922b, cb0aef9c42b54f00a100de5e833e85e9; no voters:
I20250109 21:08:47.544149 32491 raft_consensus.cc:2798] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:47.544332 32491 raft_consensus.cc:695] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [term 1 LEADER]: Becoming Leader. State: Replica: cb0aef9c42b54f00a100de5e833e85e9, State: Running, Role: LEADER
I20250109 21:08:47.544656 32491 consensus_queue.cc:237] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa346abef3745939a9bacbcac7898d9" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43503 } } peers { permanent_uuid: "cb0aef9c42b54f00a100de5e833e85e9" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 41431 } } peers { permanent_uuid: "437cbabc1ed5447a9a555c1b666c922b" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 46353 } }
I20250109 21:08:47.547608 32004 catalog_manager.cc:5526] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb0aef9c42b54f00a100de5e833e85e9 (127.12.126.66). New cstate: current_term: 1 leader_uuid: "cb0aef9c42b54f00a100de5e833e85e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa346abef3745939a9bacbcac7898d9" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb0aef9c42b54f00a100de5e833e85e9" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 41431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "437cbabc1ed5447a9a555c1b666c922b" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 46353 } health_report { overall_health: UNKNOWN } } }
I20250109 21:08:47.567559 32481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.564903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45534 (local address 127.12.126.66:41431)
0109 21:08:47.565025 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:47.565028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:47.565040 (+ 12us) server_negotiation.cc:407] Connection header received
0109 21:08:47.565109 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:47.565111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:47.565161 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:47.565253 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:08:47.565509 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.566036 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.566583 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.566713 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.566823 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:08:47.567256 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:08:47.567347 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:47.567413 (+ 66us) server_negotiation.cc:299] Negotiation successful
0109 21:08:47.567457 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250109 21:08:47.572803 32420 raft_consensus.cc:1270] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9 [term 1 FOLLOWER]: Refusing update from remote peer cb0aef9c42b54f00a100de5e833e85e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:08:47.572808 32152 raft_consensus.cc:1270] T 0a5cdb16277648b3921d739b20bed535 P 437cbabc1ed5447a9a555c1b666c922b [term 1 FOLLOWER]: Refusing update from remote peer cb0aef9c42b54f00a100de5e833e85e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:08:47.573428 32493 consensus_queue.cc:1035] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fa346abef3745939a9bacbcac7898d9" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 43503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:08:47.573583 32491 consensus_queue.cc:1035] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "437cbabc1ed5447a9a555c1b666c922b" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 46353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:08:47.579221 32496 mvcc.cc:204] Tried to move back new op lower bound from 7112527575331741696 to 7112527575225753600. Current Snapshot: MvccSnapshot[applied={T|T < 7112527575331741696}]
I20250109 21:08:47.580199 32497 mvcc.cc:204] Tried to move back new op lower bound from 7112527575331741696 to 7112527575225753600. Current Snapshot: MvccSnapshot[applied={T|T < 7112527575331741696}]
I20250109 21:08:47.587448 32006 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38134:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:08:47.587586 32006 catalog_manager.cc:2646] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38134:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:08:47.589829 32006 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 2dc6fd7dc156469592014695dc4b87ba: Sending DeleteTablet for 3 replicas of tablet 0a5cdb16277648b3921d739b20bed535
I20250109 21:08:47.590560 32266 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5cdb16277648b3921d739b20bed535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:47 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:45532
I20250109 21:08:47.590641 32132 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5cdb16277648b3921d739b20bed535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:47 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:56174
I20250109 21:08:47.590754 32400 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5cdb16277648b3921d739b20bed535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:47 UTC) from {username='kudu', principal='kudu/127.12.126.126@KRBTEST.COM'} at 127.0.0.1:56530
I20250109 21:08:47.591318 32506 tablet_replica.cc:331] T 0a5cdb16277648b3921d739b20bed535 P 437cbabc1ed5447a9a555c1b666c922b: stopping tablet replica
I20250109 21:08:47.591586 32506 raft_consensus.cc:2238] T 0a5cdb16277648b3921d739b20bed535 P 437cbabc1ed5447a9a555c1b666c922b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:08:47.591784 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 32068
I20250109 21:08:47.591908 32506 raft_consensus.cc:2267] T 0a5cdb16277648b3921d739b20bed535 P 437cbabc1ed5447a9a555c1b666c922b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:47.592226 32507 tablet_replica.cc:331] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9: stopping tablet replica
I20250109 21:08:47.592458 32507 raft_consensus.cc:2238] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:08:47.592696 32507 raft_consensus.cc:2267] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:47.593024 32505 tablet_replica.cc:331] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9: stopping tablet replica
I20250109 21:08:47.593225 32505 raft_consensus.cc:2238] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [term 1 LEADER]: Raft consensus shutting down.
I20250109 21:08:47.593555 32505 raft_consensus.cc:2267] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:47.595381 32507 ts_tablet_manager.cc:1905] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:08:47.595789 32505 ts_tablet_manager.cc:1905] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:08:47.598321 32505 ts_tablet_manager.cc:1918] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:08:47.598484 32505 log.cc:1198] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/0a5cdb16277648b3921d739b20bed535
I20250109 21:08:47.598495 32507 ts_tablet_manager.cc:1918] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:08:47.598668 32507 log.cc:1198] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/0a5cdb16277648b3921d739b20bed535
I20250109 21:08:47.598920 32505 ts_tablet_manager.cc:1939] T 0a5cdb16277648b3921d739b20bed535 P cb0aef9c42b54f00a100de5e833e85e9: Deleting consensus metadata
I20250109 21:08:47.599090 32507 ts_tablet_manager.cc:1939] T 0a5cdb16277648b3921d739b20bed535 P 1fa346abef3745939a9bacbcac7898d9: Deleting consensus metadata
I20250109 21:08:47.600169 31993 catalog_manager.cc:4872] TS cb0aef9c42b54f00a100de5e833e85e9 (127.12.126.66:41431): tablet 0a5cdb16277648b3921d739b20bed535 (table test-table [id=12a5f996db894a9aa10eb9bc02be858e]) successfully deleted
I20250109 21:08:47.600212 31990 catalog_manager.cc:4872] TS 1fa346abef3745939a9bacbcac7898d9 (127.12.126.67:43503): tablet 0a5cdb16277648b3921d739b20bed535 (table test-table [id=12a5f996db894a9aa10eb9bc02be858e]) successfully deleted
W20250109 21:08:47.600664 31992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.126.65:46353 (error 108)
I20250109 21:08:47.601423 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 32203
I20250109 21:08:47.601670 32480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.601353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:46353 (local address 127.0.0.1:56180)
0109 21:08:47.601499 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:47.601570 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.65:46353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20250109 21:08:47.603101 31992 catalog_manager.cc:4615] TS 437cbabc1ed5447a9a555c1b666c922b (127.12.126.65:46353): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a5cdb16277648b3921d739b20bed535: Network error: Client connection negotiation failed: client connection to 127.12.126.65:46353: connect: Connection refused (error 111)
I20250109 21:08:47.607739 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 32337
I20250109 21:08:47.613710 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 31974
2025-01-09T21:08:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2077 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2025-01-09T21:08:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:08:47Z Disabled control of system clock
I20250109 21:08:47.642428 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:38783
--webserver_interface=127.12.126.126
--webserver_port=0
--builtin_ntp_servers=127.12.126.84:39573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:38783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20250109 21:08:47.743767 32512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:47.743983 32512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:47.744028 32512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:47.747020 32512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:08:47.747071 32512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:47.747087 32512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:47.747104 32512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:08:47.747126 32512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:08:47.750661 32512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:38783
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:38783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:47.751582 32512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:47.752614 32512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:47.759331 32518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:47.759975 32520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:47.760561 32517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:47.761445 32512 server_base.cc:1034] running on GCE node
I20250109 21:08:47.761914 32512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:47.762751 32512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:47.763922 32512 hybrid_clock.cc:648] HybridClock initialized: now 1736456927763898 us; error 37 us; skew 500 ppm
I20250109 21:08:47.765633 32512 webserver.cc:458] Webserver started at http://127.12.126.126:42661/ using document root <none> and password file <none>
I20250109 21:08:47.766167 32512 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:47.766228 32512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:47.766417 32512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:47.768121 32512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "1f189e97b3614de79c5bf99657c54c74"
format_stamp: "Formatted at 2025-01-09 21:08:47 on dist-test-slave-g0zb"
server_key: "063564a538fb6c01f9518cd25b14d08d"
server_key_iv: "951275ba4b3ef9ecb5d69b01636360d1"
server_key_version: "encryptionkey@0"
I20250109 21:08:47.768566 32512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "1f189e97b3614de79c5bf99657c54c74"
format_stamp: "Formatted at 2025-01-09 21:08:47 on dist-test-slave-g0zb"
server_key: "063564a538fb6c01f9518cd25b14d08d"
server_key_iv: "951275ba4b3ef9ecb5d69b01636360d1"
server_key_version: "encryptionkey@0"
I20250109 21:08:47.771616 32512 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:08:47.773650 32526 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:47.774569 32512 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:47.774672 32512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "1f189e97b3614de79c5bf99657c54c74"
format_stamp: "Formatted at 2025-01-09 21:08:47 on dist-test-slave-g0zb"
server_key: "063564a538fb6c01f9518cd25b14d08d"
server_key_iv: "951275ba4b3ef9ecb5d69b01636360d1"
server_key_version: "encryptionkey@0"
I20250109 21:08:47.774758 32512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:47.805918 32512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:47.806521 32512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:47.806672 32512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:47.812691 32512 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:38783
I20250109 21:08:47.812703 32578 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:38783 every 8 connection(s)
I20250109 21:08:47.813587 32512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:08:47.816062 32579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:47.817102 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 32512
I20250109 21:08:47.817199 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:08:47.817433 12793 external_mini_cluster.cc:1453] Setting key 2c1f4e8f12d1462bd37ba6f8713efaa7
I20250109 21:08:47.821493 32579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74: Bootstrap starting.
I20250109 21:08:47.823562 32579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:47.824360 32579 log.cc:826] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:47.824728 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:47.818683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.12.126.126:38783)
0109 21:08:47.819314 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:47.819324 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:47.819356 (+ 32us) server_negotiation.cc:407] Connection header received
0109 21:08:47.819845 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:47.819860 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:47.820179 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:47.820466 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:47.820955 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.821722 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:47.822353 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:47.822616 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:47.823150 (+ 534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:47.823166 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:47.823175 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:47.823483 (+ 308us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:47.823518 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:47.823531 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:47.823620 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:47.823804 (+ 184us) server_negotiation.cc:299] Negotiation successful
0109 21:08:47.824034 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":137,"threads_started":1}
I20250109 21:08:47.825987 32579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74: No bootstrap required, opened a new log
I20250109 21:08:47.828979 32579 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f189e97b3614de79c5bf99657c54c74" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38783 } }
I20250109 21:08:47.829145 32579 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:47.829208 32579 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f189e97b3614de79c5bf99657c54c74, State: Initialized, Role: FOLLOWER
I20250109 21:08:47.829560 32579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f189e97b3614de79c5bf99657c54c74" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38783 } }
I20250109 21:08:47.829674 32579 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:08:47.829734 32579 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:08:47.829843 32579 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:47.830535 32579 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f189e97b3614de79c5bf99657c54c74" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38783 } }
I20250109 21:08:47.830878 32579 leader_election.cc:304] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f189e97b3614de79c5bf99657c54c74; no voters:
I20250109 21:08:47.831132 32579 leader_election.cc:290] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:08:47.831213 32584 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:47.831394 32584 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74 [term 1 LEADER]: Becoming Leader. State: Replica: 1f189e97b3614de79c5bf99657c54c74, State: Running, Role: LEADER
I20250109 21:08:47.831707 32584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f189e97b3614de79c5bf99657c54c74" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38783 } }
I20250109 21:08:47.832055 32579 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:08:47.833189 32586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f189e97b3614de79c5bf99657c54c74. Latest consensus state: current_term: 1 leader_uuid: "1f189e97b3614de79c5bf99657c54c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f189e97b3614de79c5bf99657c54c74" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38783 } } }
I20250109 21:08:47.833339 32586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:47.833197 32585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f189e97b3614de79c5bf99657c54c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f189e97b3614de79c5bf99657c54c74" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 38783 } } }
I20250109 21:08:47.833448 32585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:47.833729 32593 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:08:47.836339 32593 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:08:47.840902 32593 catalog_manager.cc:1348] Generated new cluster ID: 8e466f84c86f45cd8fa40260ce0ab5d9
I20250109 21:08:47.840960 32593 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:08:47.868876 32593 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:08:47.870059 32593 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:08:47.886082 32593 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74: Generated new TSK 0
I20250109 21:08:47.886778 32593 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:08:47.895679 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.65:0
--local_ip_for_outbound_sockets=127.12.126.65
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:38783
--builtin_ntp_servers=127.12.126.84:39573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20250109 21:08:48.001468 32603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:48.001716 32603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:48.001806 32603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:48.004947 32603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:48.005012 32603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:48.005120 32603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.65
I20250109 21:08:48.008800 32603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.126.65
--webserver_port=0
--tserver_master_addrs=127.12.126.126:38783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.65
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:48.009878 32603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:48.011098 32603 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:48.019017 32608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:48.019759 32611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:48.020498 32609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:48.021229 32603 server_base.cc:1034] running on GCE node
I20250109 21:08:48.021561 32603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:48.022104 32603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:48.023255 32603 hybrid_clock.cc:648] HybridClock initialized: now 1736456928023236 us; error 45 us; skew 500 ppm
I20250109 21:08:48.025094 32603 webserver.cc:458] Webserver started at http://127.12.126.65:43655/ using document root <none> and password file <none>
I20250109 21:08:48.025645 32603 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:48.025705 32603 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:48.025876 32603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:48.027913 32603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-0/data/instance:
uuid: "7dc83f9bb94140bd801a9df8940002c4"
format_stamp: "Formatted at 2025-01-09 21:08:48 on dist-test-slave-g0zb"
server_key: "af4d8b75c83beb571fa79f26604f8a80"
server_key_iv: "43f546a41b86fe1db5b10e83ba47c147"
server_key_version: "encryptionkey@0"
I20250109 21:08:48.028438 32603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance:
uuid: "7dc83f9bb94140bd801a9df8940002c4"
format_stamp: "Formatted at 2025-01-09 21:08:48 on dist-test-slave-g0zb"
server_key: "af4d8b75c83beb571fa79f26604f8a80"
server_key_iv: "43f546a41b86fe1db5b10e83ba47c147"
server_key_version: "encryptionkey@0"
I20250109 21:08:48.032078 32603 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:08:48.034106 32617 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:48.035081 32603 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:08:48.035180 32603 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-0/wal
uuid: "7dc83f9bb94140bd801a9df8940002c4"
format_stamp: "Formatted at 2025-01-09 21:08:48 on dist-test-slave-g0zb"
server_key: "af4d8b75c83beb571fa79f26604f8a80"
server_key_iv: "43f546a41b86fe1db5b10e83ba47c147"
server_key_version: "encryptionkey@0"
I20250109 21:08:48.035261 32603 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:48.053946 32603 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:48.054569 32603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:48.054751 32603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:48.055315 32603 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:48.056190 32603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:48.056239 32603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:48.056300 32603 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:48.056331 32603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:48.064621 32603 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.65:34567
I20250109 21:08:48.064646 32730 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.65:34567 every 8 connection(s)
I20250109 21:08:48.065536 32603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-0/data/info.pb
I20250109 21:08:48.071518 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 32603
I20250109 21:08:48.071641 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-0/wal/instance
I20250109 21:08:48.072007 12793 external_mini_cluster.cc:1453] Setting key 8567a15fe211c17d358db50c4a65a0aa
I20250109 21:08:48.073274 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.067286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:56439 (local address 127.12.126.126:38783)
0109 21:08:48.067434 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:48.067438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:48.068353 (+ 915us) server_negotiation.cc:407] Connection header received
0109 21:08:48.069128 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:48.069132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:48.069194 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:48.069275 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:48.070597 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.071144 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:48.071866 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.072101 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:48.072818 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:48.072828 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:48.072831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:48.072841 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:48.072861 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:48.072868 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:48.072928 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:48.073096 (+ 168us) server_negotiation.cc:299] Negotiation successful
0109 21:08:48.073153 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250109 21:08:48.074080 32733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.067567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:38783 (local address 127.12.126.65:56439)
0109 21:08:48.068211 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:48.068243 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:48.068921 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:48.069454 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:48.069461 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:48.069532 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:48.070460 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:48.070473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.071261 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:48.071264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:48.071697 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:48.071709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.071929 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:48.072594 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:08:48.072596 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:48.072629 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:08:48.072683 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:48.072944 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:48.072952 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:48.072958 (+ 6us) client_negotiation.cc:769] Sending connection context
0109 21:08:48.073102 (+ 144us) client_negotiation.cc:240] Negotiation successful
0109 21:08:48.073332 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":111,"threads_started":1}
I20250109 21:08:48.074146 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.66:0
--local_ip_for_outbound_sockets=127.12.126.66
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:38783
--builtin_ntp_servers=127.12.126.84:39573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20250109 21:08:48.075860 32731 heartbeater.cc:346] Connected to a master server at 127.12.126.126:38783
I20250109 21:08:48.076083 32731 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:48.076700 32731 heartbeater.cc:510] Master 127.12.126.126:38783 requested a full tablet report, sending...
I20250109 21:08:48.078152 32543 ts_manager.cc:194] Registered new tserver with Master: 7dc83f9bb94140bd801a9df8940002c4 (127.12.126.65:34567)
I20250109 21:08:48.079376 32543 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.126.65:56439
W20250109 21:08:48.183248 32734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:48.183486 32734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:48.183547 32734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:48.186718 32734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:48.186786 32734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:48.186925 32734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.66
I20250109 21:08:48.190693 32734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.126.66
--webserver_port=0
--tserver_master_addrs=127.12.126.126:38783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.66
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:48.191835 32734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:48.192968 32734 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:48.201064 32742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:48.201809 32739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:48.202770 32740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:48.203563 32734 server_base.cc:1034] running on GCE node
I20250109 21:08:48.203959 32734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:48.204535 32734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:48.205693 32734 hybrid_clock.cc:648] HybridClock initialized: now 1736456928205668 us; error 40 us; skew 500 ppm
I20250109 21:08:48.207670 32734 webserver.cc:458] Webserver started at http://127.12.126.66:42881/ using document root <none> and password file <none>
I20250109 21:08:48.208235 32734 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:48.208297 32734 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:48.208468 32734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:48.210291 32734 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-1/data/instance:
uuid: "c43d71d6bba449908f6df0da74ab07c1"
format_stamp: "Formatted at 2025-01-09 21:08:48 on dist-test-slave-g0zb"
server_key: "8e305453d2dd9b25c10a5769d6db30b5"
server_key_iv: "af3e2d12085e900d8cf682df1f15c788"
server_key_version: "encryptionkey@0"
I20250109 21:08:48.210759 32734 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance:
uuid: "c43d71d6bba449908f6df0da74ab07c1"
format_stamp: "Formatted at 2025-01-09 21:08:48 on dist-test-slave-g0zb"
server_key: "8e305453d2dd9b25c10a5769d6db30b5"
server_key_iv: "af3e2d12085e900d8cf682df1f15c788"
server_key_version: "encryptionkey@0"
I20250109 21:08:48.214100 32734 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250109 21:08:48.216341 32748 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:48.217344 32734 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:08:48.217451 32734 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-1/wal
uuid: "c43d71d6bba449908f6df0da74ab07c1"
format_stamp: "Formatted at 2025-01-09 21:08:48 on dist-test-slave-g0zb"
server_key: "8e305453d2dd9b25c10a5769d6db30b5"
server_key_iv: "af3e2d12085e900d8cf682df1f15c788"
server_key_version: "encryptionkey@0"
I20250109 21:08:48.217537 32734 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:48.247498 32734 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:48.248139 32734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:48.248313 32734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:48.248860 32734 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:48.249774 32734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:48.249833 32734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:48.249899 32734 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:48.249928 32734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:48.258857 32734 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.66:43029
I20250109 21:08:48.258888 393 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.66:43029 every 8 connection(s)
I20250109 21:08:48.259809 32734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-1/data/info.pb
I20250109 21:08:48.260449 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 32734
I20250109 21:08:48.260566 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-1/wal/instance
I20250109 21:08:48.260833 12793 external_mini_cluster.cc:1453] Setting key a41a7e79f8f7b10feb207d43fcf11a9f
I20250109 21:08:48.263077 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.126.67:0
--local_ip_for_outbound_sockets=127.12.126.67
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:38783
--builtin_ntp_servers=127.12.126.84:39573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20250109 21:08:48.268847 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.261643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.66:52915 (local address 127.12.126.126:38783)
0109 21:08:48.261800 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:48.261804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:48.262938 (+ 1134us) server_negotiation.cc:407] Connection header received
0109 21:08:48.264115 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:48.264119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:48.264167 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:48.264243 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:48.265679 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.266202 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:48.267156 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.267304 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:48.268292 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:48.268311 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:48.268313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:48.268326 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:48.268346 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:48.268354 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:48.268422 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:48.268653 (+ 231us) server_negotiation.cc:299] Negotiation successful
0109 21:08:48.268732 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250109 21:08:48.269765 396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.262020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:38783 (local address 127.12.126.66:52915)
0109 21:08:48.262751 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:48.262836 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:48.263851 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:48.264431 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:48.264444 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:48.264558 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:48.265540 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:48.265553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.266335 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:48.266341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:48.267000 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:48.267016 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.267273 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:48.267983 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:08:48.267986 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:48.268046 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:08:48.268125 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:48.268445 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:48.268455 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:48.268463 (+ 8us) client_negotiation.cc:769] Sending connection context
0109 21:08:48.268622 (+ 159us) client_negotiation.cc:240] Negotiation successful
0109 21:08:48.268887 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":121,"threads_started":1}
I20250109 21:08:48.270985 394 heartbeater.cc:346] Connected to a master server at 127.12.126.126:38783
I20250109 21:08:48.271202 394 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:48.271723 394 heartbeater.cc:510] Master 127.12.126.126:38783 requested a full tablet report, sending...
I20250109 21:08:48.272782 32543 ts_manager.cc:194] Registered new tserver with Master: c43d71d6bba449908f6df0da74ab07c1 (127.12.126.66:43029)
I20250109 21:08:48.273597 32543 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.126.66:52915
W20250109 21:08:48.371999 397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:48.372233 397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:48.372283 397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:48.375382 397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:48.375439 397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:48.375512 397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.126.67
I20250109 21:08:48.379336 397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.126.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.126.67
--webserver_port=0
--tserver_master_addrs=127.12.126.126:38783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.126.67
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:48.380375 397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:48.381461 397 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250109 21:08:48.390549 397 server_base.cc:1034] running on GCE node
W20250109 21:08:48.390673 403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:48.391391 405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:48.391971 402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:48.392388 397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:48.392931 397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:48.394074 397 hybrid_clock.cc:648] HybridClock initialized: now 1736456928394052 us; error 42 us; skew 500 ppm
I20250109 21:08:48.395978 397 webserver.cc:458] Webserver started at http://127.12.126.67:39491/ using document root <none> and password file <none>
I20250109 21:08:48.396545 397 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:48.396600 397 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:48.396780 397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:48.398561 397 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-2/data/instance:
uuid: "0c1cd9e415124aefa0f02d4b11002aa4"
format_stamp: "Formatted at 2025-01-09 21:08:48 on dist-test-slave-g0zb"
server_key: "4335c7b9851fd28c968331c1905f2e44"
server_key_iv: "e7d7cc66512dbede0646fb098242d168"
server_key_version: "encryptionkey@0"
I20250109 21:08:48.399066 397 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance:
uuid: "0c1cd9e415124aefa0f02d4b11002aa4"
format_stamp: "Formatted at 2025-01-09 21:08:48 on dist-test-slave-g0zb"
server_key: "4335c7b9851fd28c968331c1905f2e44"
server_key_iv: "e7d7cc66512dbede0646fb098242d168"
server_key_version: "encryptionkey@0"
I20250109 21:08:48.402390 397 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20250109 21:08:48.404515 411 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:48.405561 397 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:08:48.405663 397 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-2/wal
uuid: "0c1cd9e415124aefa0f02d4b11002aa4"
format_stamp: "Formatted at 2025-01-09 21:08:48 on dist-test-slave-g0zb"
server_key: "4335c7b9851fd28c968331c1905f2e44"
server_key_iv: "e7d7cc66512dbede0646fb098242d168"
server_key_version: "encryptionkey@0"
I20250109 21:08:48.405769 397 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:48.420737 397 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:48.421309 397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:48.421458 397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:48.422045 397 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 21:08:48.422915 397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 21:08:48.422961 397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:48.422999 397 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 21:08:48.423020 397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:48.430961 397 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.67:40209
I20250109 21:08:48.430994 524 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.67:40209 every 8 connection(s)
I20250109 21:08:48.431838 397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-2/data/info.pb
I20250109 21:08:48.439548 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.433576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.67:54847 (local address 127.12.126.126:38783)
0109 21:08:48.433782 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:48.433790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:48.434647 (+ 857us) server_negotiation.cc:407] Connection header received
0109 21:08:48.435465 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:48.435468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:48.435530 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:48.435609 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:48.436944 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.437444 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:48.438065 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.438244 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:48.439050 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:48.439072 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:48.439075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:48.439086 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:48.439104 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:48.439111 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:48.439177 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:48.439322 (+ 145us) server_negotiation.cc:299] Negotiation successful
0109 21:08:48.439403 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20250109 21:08:48.439782 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 397
I20250109 21:08:48.439888 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-2/wal/instance
I20250109 21:08:48.440173 527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.433904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:38783 (local address 127.12.126.67:54847)
0109 21:08:48.434510 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:48.434545 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:48.435264 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:48.435780 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:48.435788 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:48.435863 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:08:48.436800 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:48.436814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.437561 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:48.437564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:48.437952 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:48.437959 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.438166 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:48.438830 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0109 21:08:48.438832 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:08:48.438864 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0109 21:08:48.438917 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:08:48.439187 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:08:48.439196 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:08:48.439203 (+ 7us) client_negotiation.cc:769] Sending connection context
0109 21:08:48.439320 (+ 117us) client_negotiation.cc:240] Negotiation successful
0109 21:08:48.439532 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
I20250109 21:08:48.440178 12793 external_mini_cluster.cc:1453] Setting key 691fed93af35f8a6bca91bebba75046e
I20250109 21:08:48.441357 525 heartbeater.cc:346] Connected to a master server at 127.12.126.126:38783
I20250109 21:08:48.441576 525 heartbeater.cc:463] Registering TS with master...
I20250109 21:08:48.442168 525 heartbeater.cc:510] Master 127.12.126.126:38783 requested a full tablet report, sending...
I20250109 21:08:48.443085 32542 ts_manager.cc:194] Registered new tserver with Master: 0c1cd9e415124aefa0f02d4b11002aa4 (127.12.126.67:40209)
I20250109 21:08:48.443635 32542 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.126.67:54847
I20250109 21:08:48.453663 12793 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 21:08:48.458593 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.455493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38638 (local address 127.12.126.126:38783)
0109 21:08:48.455648 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:48.455653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:48.455775 (+ 122us) server_negotiation.cc:407] Connection header received
0109 21:08:48.455864 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:48.455867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:48.455908 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:48.455977 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:48.456324 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.456811 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:48.457414 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.457530 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:48.458206 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:48.458220 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:48.458222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:48.458233 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:48.458252 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:48.458259 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:48.458325 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:48.458392 (+ 67us) server_negotiation.cc:299] Negotiation successful
0109 21:08:48.458432 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250109 21:08:48.461432 32542 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 21:08:48.463974 32542 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 21:08:48.476404 537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.471760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:40209 (local address 127.0.0.1:57272)
0109 21:08:48.472336 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:48.472350 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:48.472424 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:48.473546 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:48.473549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:48.473567 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:48.473799 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:48.473807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.475186 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:48.475192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:48.475991 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:48.476004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.476110 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:48.476128 (+ 18us) client_negotiation.cc:769] Sending connection context
0109 21:08:48.476177 (+ 49us) client_negotiation.cc:240] Negotiation successful
0109 21:08:48.476240 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":102,"threads_started":1}
I20250109 21:08:48.477017 538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.471727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.12.126.65:34567)
0109 21:08:48.472550 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:48.472557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:48.472582 (+ 25us) server_negotiation.cc:407] Connection header received
0109 21:08:48.472676 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:48.472685 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:48.472896 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:48.473624 (+ 728us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:48.473760 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.474926 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:48.476054 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.476572 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:48.476727 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:48.476796 (+ 69us) server_negotiation.cc:299] Negotiation successful
0109 21:08:48.476880 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":136,"threads_started":1}
I20250109 21:08:48.477176 541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.472488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57732 (local address 127.12.126.66:43029)
0109 21:08:48.472881 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:48.472888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:48.473197 (+ 309us) server_negotiation.cc:407] Connection header received
0109 21:08:48.473297 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:48.473303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:48.473470 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:48.473623 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:48.474065 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.474763 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:48.476312 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.476805 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:48.476915 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:48.476973 (+ 58us) server_negotiation.cc:299] Negotiation successful
0109 21:08:48.477035 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":125,"threads_started":1}
I20250109 21:08:48.476406 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.471563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:34567 (local address 127.0.0.1:53850)
0109 21:08:48.472052 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:48.472090 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:48.472226 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:48.473120 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:48.473129 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:48.473151 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:48.473463 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:48.473471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.475048 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:48.475051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:48.475888 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:48.475894 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.476004 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:48.476055 (+ 51us) client_negotiation.cc:769] Sending connection context
0109 21:08:48.476150 (+ 95us) client_negotiation.cc:240] Negotiation successful
0109 21:08:48.476242 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":78,"threads_started":1}
I20250109 21:08:48.477433 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.472205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:43029 (local address 127.0.0.1:57732)
0109 21:08:48.472951 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:48.472978 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:48.473090 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:48.473728 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:48.473731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:48.473746 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:48.473943 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:48.473949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.474923 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:48.474927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:48.475878 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:48.475889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.476003 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:48.476055 (+ 52us) client_negotiation.cc:769] Sending connection context
0109 21:08:48.476150 (+ 95us) client_negotiation.cc:240] Negotiation successful
0109 21:08:48.476242 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":83,"threads_started":1}
I20250109 21:08:48.477550 540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.472422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57272 (local address 127.12.126.67:40209)
0109 21:08:48.473127 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:48.473135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:48.473162 (+ 27us) server_negotiation.cc:407] Connection header received
0109 21:08:48.473251 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:48.473268 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:48.473412 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:48.473568 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:48.473929 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.475084 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:48.476598 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.477091 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:48.477240 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:48.477315 (+ 75us) server_negotiation.cc:299] Negotiation successful
0109 21:08:48.477400 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":149,"threads_started":1}
I20250109 21:08:48.479903 459 tablet_service.cc:1467] Processing CreateTablet for tablet cdb3b27087ad4d5cacaf2744cdcf6c06 (DEFAULT_TABLE table=test-table [id=80e4cc2833de4a43ac71fea3e3e904c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:48.479967 32665 tablet_service.cc:1467] Processing CreateTablet for tablet cdb3b27087ad4d5cacaf2744cdcf6c06 (DEFAULT_TABLE table=test-table [id=80e4cc2833de4a43ac71fea3e3e904c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:48.479954 328 tablet_service.cc:1467] Processing CreateTablet for tablet cdb3b27087ad4d5cacaf2744cdcf6c06 (DEFAULT_TABLE table=test-table [id=80e4cc2833de4a43ac71fea3e3e904c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 21:08:48.480736 459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb3b27087ad4d5cacaf2744cdcf6c06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:48.480819 32665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb3b27087ad4d5cacaf2744cdcf6c06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:48.480819 328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb3b27087ad4d5cacaf2744cdcf6c06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:48.486040 542 tablet_bootstrap.cc:492] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4: Bootstrap starting.
I20250109 21:08:48.487164 543 tablet_bootstrap.cc:492] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4: Bootstrap starting.
I20250109 21:08:48.487510 544 tablet_bootstrap.cc:492] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1: Bootstrap starting.
I20250109 21:08:48.489192 544 tablet_bootstrap.cc:654] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:48.489266 543 tablet_bootstrap.cc:654] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:48.489266 542 tablet_bootstrap.cc:654] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:48.490080 544 log.cc:826] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:48.490128 542 log.cc:826] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:48.490250 543 log.cc:826] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:48.491931 543 tablet_bootstrap.cc:492] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4: No bootstrap required, opened a new log
I20250109 21:08:48.491931 544 tablet_bootstrap.cc:492] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1: No bootstrap required, opened a new log
I20250109 21:08:48.492048 542 tablet_bootstrap.cc:492] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4: No bootstrap required, opened a new log
I20250109 21:08:48.492110 544 ts_tablet_manager.cc:1397] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250109 21:08:48.492110 543 ts_tablet_manager.cc:1397] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250109 21:08:48.492223 542 ts_tablet_manager.cc:1397] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20250109 21:08:48.494854 542 raft_consensus.cc:357] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc83f9bb94140bd801a9df8940002c4" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34567 } } peers { permanent_uuid: "0c1cd9e415124aefa0f02d4b11002aa4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40209 } } peers { permanent_uuid: "c43d71d6bba449908f6df0da74ab07c1" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43029 } }
I20250109 21:08:48.495038 542 raft_consensus.cc:383] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:48.495093 542 raft_consensus.cc:738] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1cd9e415124aefa0f02d4b11002aa4, State: Initialized, Role: FOLLOWER
I20250109 21:08:48.495350 543 raft_consensus.cc:357] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc83f9bb94140bd801a9df8940002c4" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34567 } } peers { permanent_uuid: "0c1cd9e415124aefa0f02d4b11002aa4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40209 } } peers { permanent_uuid: "c43d71d6bba449908f6df0da74ab07c1" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43029 } }
I20250109 21:08:48.495350 544 raft_consensus.cc:357] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc83f9bb94140bd801a9df8940002c4" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34567 } } peers { permanent_uuid: "0c1cd9e415124aefa0f02d4b11002aa4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40209 } } peers { permanent_uuid: "c43d71d6bba449908f6df0da74ab07c1" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43029 } }
I20250109 21:08:48.495548 544 raft_consensus.cc:383] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:48.495548 543 raft_consensus.cc:383] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:48.495487 542 consensus_queue.cc:260] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc83f9bb94140bd801a9df8940002c4" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34567 } } peers { permanent_uuid: "0c1cd9e415124aefa0f02d4b11002aa4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40209 } } peers { permanent_uuid: "c43d71d6bba449908f6df0da74ab07c1" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43029 } }
I20250109 21:08:48.495607 543 raft_consensus.cc:738] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dc83f9bb94140bd801a9df8940002c4, State: Initialized, Role: FOLLOWER
I20250109 21:08:48.495607 544 raft_consensus.cc:738] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c43d71d6bba449908f6df0da74ab07c1, State: Initialized, Role: FOLLOWER
I20250109 21:08:48.496028 544 consensus_queue.cc:260] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc83f9bb94140bd801a9df8940002c4" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34567 } } peers { permanent_uuid: "0c1cd9e415124aefa0f02d4b11002aa4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40209 } } peers { permanent_uuid: "c43d71d6bba449908f6df0da74ab07c1" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43029 } }
I20250109 21:08:48.496028 543 consensus_queue.cc:260] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc83f9bb94140bd801a9df8940002c4" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34567 } } peers { permanent_uuid: "0c1cd9e415124aefa0f02d4b11002aa4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40209 } } peers { permanent_uuid: "c43d71d6bba449908f6df0da74ab07c1" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43029 } }
I20250109 21:08:48.496338 542 ts_tablet_manager.cc:1428] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20250109 21:08:48.496721 525 heartbeater.cc:502] Master 127.12.126.126:38783 was elected leader, sending a full tablet report...
I20250109 21:08:48.496733 32731 heartbeater.cc:502] Master 127.12.126.126:38783 was elected leader, sending a full tablet report...
I20250109 21:08:48.496855 543 ts_tablet_manager.cc:1428] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250109 21:08:48.497153 394 heartbeater.cc:502] Master 127.12.126.126:38783 was elected leader, sending a full tablet report...
I20250109 21:08:48.497931 544 ts_tablet_manager.cc:1428] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20250109 21:08:48.510685 395 tablet.cc:2367] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250109 21:08:48.566602 32732 tablet.cc:2367] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250109 21:08:48.682723 526 tablet.cc:2367] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:08:48.706645 550 raft_consensus.cc:491] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 21:08:48.706907 550 raft_consensus.cc:513] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc83f9bb94140bd801a9df8940002c4" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34567 } } peers { permanent_uuid: "0c1cd9e415124aefa0f02d4b11002aa4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40209 } } peers { permanent_uuid: "c43d71d6bba449908f6df0da74ab07c1" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43029 } }
I20250109 21:08:48.707981 550 leader_election.cc:290] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c1cd9e415124aefa0f02d4b11002aa4 (127.12.126.67:40209), c43d71d6bba449908f6df0da74ab07c1 (127.12.126.66:43029)
I20250109 21:08:48.712141 552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.708632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.66:43029 (local address 127.12.126.65:47725)
0109 21:08:48.708870 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:48.708883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:48.709002 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:48.709507 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:48.709510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:48.709527 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:48.709779 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:48.709785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.710850 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:48.710853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:48.711694 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:48.711704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.711823 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:48.711846 (+ 23us) client_negotiation.cc:769] Sending connection context
0109 21:08:48.711901 (+ 55us) client_negotiation.cc:240] Negotiation successful
0109 21:08:48.711960 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"spinlock_wait_cycles":1408,"thread_start_us":64,"threads_started":1}
I20250109 21:08:48.712378 540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.708309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:42027 (local address 127.12.126.67:40209)
0109 21:08:48.708456 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:48.708462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:48.708625 (+ 163us) server_negotiation.cc:407] Connection header received
0109 21:08:48.708782 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:48.708787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:48.708834 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:48.708905 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:48.709394 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.710206 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:48.711471 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.711928 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:48.711964 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:48.712177 (+ 213us) server_negotiation.cc:299] Negotiation successful
0109 21:08:48.712253 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20250109 21:08:48.712669 541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.709052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.126.65:47725 (local address 127.12.126.66:43029)
0109 21:08:48.709206 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:48.709211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:48.709224 (+ 13us) server_negotiation.cc:407] Connection header received
0109 21:08:48.709272 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:48.709275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:48.709331 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:48.709403 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 21:08:48.709912 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.710686 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:48.711834 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.712365 (+ 531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:48.712400 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:48.712461 (+ 61us) server_negotiation.cc:299] Negotiation successful
0109 21:08:48.712520 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20250109 21:08:48.712766 551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.708227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.67:40209 (local address 127.12.126.65:42027)
0109 21:08:48.708535 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:48.708553 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0109 21:08:48.708679 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:08:48.708924 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:08:48.708927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:08:48.708946 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0109 21:08:48.709252 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:48.709261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.710347 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:08:48.710350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:08:48.711331 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:08:48.711340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.712030 (+ 690us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:48.712044 (+ 14us) client_negotiation.cc:769] Sending connection context
0109 21:08:48.712520 (+ 476us) client_negotiation.cc:240] Negotiation successful
0109 21:08:48.712594 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"spinlock_wait_cycles":14976,"thread_start_us":97,"threads_started":1}
I20250109 21:08:48.713922 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb3b27087ad4d5cacaf2744cdcf6c06" candidate_uuid: "7dc83f9bb94140bd801a9df8940002c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c43d71d6bba449908f6df0da74ab07c1" is_pre_election: true
I20250109 21:08:48.714290 348 raft_consensus.cc:2463] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7dc83f9bb94140bd801a9df8940002c4 in term 0.
I20250109 21:08:48.714820 32618 leader_election.cc:304] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7dc83f9bb94140bd801a9df8940002c4, c43d71d6bba449908f6df0da74ab07c1; no voters:
I20250109 21:08:48.715086 479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb3b27087ad4d5cacaf2744cdcf6c06" candidate_uuid: "7dc83f9bb94140bd801a9df8940002c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1cd9e415124aefa0f02d4b11002aa4" is_pre_election: true
I20250109 21:08:48.715160 550 raft_consensus.cc:2798] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 21:08:48.715237 550 raft_consensus.cc:491] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 21:08:48.715291 550 raft_consensus.cc:3054] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:48.715329 479 raft_consensus.cc:2463] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7dc83f9bb94140bd801a9df8940002c4 in term 0.
I20250109 21:08:48.716557 550 raft_consensus.cc:513] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc83f9bb94140bd801a9df8940002c4" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34567 } } peers { permanent_uuid: "0c1cd9e415124aefa0f02d4b11002aa4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40209 } } peers { permanent_uuid: "c43d71d6bba449908f6df0da74ab07c1" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43029 } }
I20250109 21:08:48.716931 550 leader_election.cc:290] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [CANDIDATE]: Term 1 election: Requested vote from peers 0c1cd9e415124aefa0f02d4b11002aa4 (127.12.126.67:40209), c43d71d6bba449908f6df0da74ab07c1 (127.12.126.66:43029)
I20250109 21:08:48.717304 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb3b27087ad4d5cacaf2744cdcf6c06" candidate_uuid: "7dc83f9bb94140bd801a9df8940002c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c43d71d6bba449908f6df0da74ab07c1"
I20250109 21:08:48.717362 479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb3b27087ad4d5cacaf2744cdcf6c06" candidate_uuid: "7dc83f9bb94140bd801a9df8940002c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1cd9e415124aefa0f02d4b11002aa4"
I20250109 21:08:48.717427 348 raft_consensus.cc:3054] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:48.717486 479 raft_consensus.cc:3054] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:48.718566 348 raft_consensus.cc:2463] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7dc83f9bb94140bd801a9df8940002c4 in term 1.
I20250109 21:08:48.718653 479 raft_consensus.cc:2463] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7dc83f9bb94140bd801a9df8940002c4 in term 1.
I20250109 21:08:48.718883 32618 leader_election.cc:304] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7dc83f9bb94140bd801a9df8940002c4, c43d71d6bba449908f6df0da74ab07c1; no voters:
I20250109 21:08:48.719071 550 raft_consensus.cc:2798] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:48.719313 550 raft_consensus.cc:695] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [term 1 LEADER]: Becoming Leader. State: Replica: 7dc83f9bb94140bd801a9df8940002c4, State: Running, Role: LEADER
I20250109 21:08:48.719676 550 consensus_queue.cc:237] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc83f9bb94140bd801a9df8940002c4" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34567 } } peers { permanent_uuid: "0c1cd9e415124aefa0f02d4b11002aa4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40209 } } peers { permanent_uuid: "c43d71d6bba449908f6df0da74ab07c1" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43029 } }
I20250109 21:08:48.723503 32542 catalog_manager.cc:5526] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7dc83f9bb94140bd801a9df8940002c4 (127.12.126.65). New cstate: current_term: 1 leader_uuid: "7dc83f9bb94140bd801a9df8940002c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc83f9bb94140bd801a9df8940002c4" member_type: VOTER last_known_addr { host: "127.12.126.65" port: 34567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c1cd9e415124aefa0f02d4b11002aa4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c43d71d6bba449908f6df0da74ab07c1" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43029 } health_report { overall_health: UNKNOWN } } }
I20250109 21:08:48.767371 538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.764456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53854 (local address 127.12.126.65:34567)
0109 21:08:48.764579 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:48.764583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:48.764596 (+ 13us) server_negotiation.cc:407] Connection header received
0109 21:08:48.764677 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:48.764679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:48.764757 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:48.764858 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:08:48.765155 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.765643 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:48.766225 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:48.766366 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:48.766456 (+ 90us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:08:48.766954 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:08:48.767050 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:48.767171 (+ 121us) server_negotiation.cc:299] Negotiation successful
0109 21:08:48.767219 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250109 21:08:48.772780 348 raft_consensus.cc:1270] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1 [term 1 FOLLOWER]: Refusing update from remote peer 7dc83f9bb94140bd801a9df8940002c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:08:48.772782 479 raft_consensus.cc:1270] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4 [term 1 FOLLOWER]: Refusing update from remote peer 7dc83f9bb94140bd801a9df8940002c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:08:48.773428 550 consensus_queue.cc:1035] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1cd9e415124aefa0f02d4b11002aa4" member_type: VOTER last_known_addr { host: "127.12.126.67" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:08:48.773619 553 consensus_queue.cc:1035] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 7dc83f9bb94140bd801a9df8940002c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c43d71d6bba449908f6df0da74ab07c1" member_type: VOTER last_known_addr { host: "127.12.126.66" port: 43029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:08:48.781865 557 mvcc.cc:204] Tried to move back new op lower bound from 7112527580246450176 to 7112527580038750208. Current Snapshot: MvccSnapshot[applied={T|T < 7112527580246450176}]
I20250109 21:08:48.782044 558 mvcc.cc:204] Tried to move back new op lower bound from 7112527580246450176 to 7112527580038750208. Current Snapshot: MvccSnapshot[applied={T|T < 7112527580246450176}]
I20250109 21:08:48.783407 556 mvcc.cc:204] Tried to move back new op lower bound from 7112527580246450176 to 7112527580038750208. Current Snapshot: MvccSnapshot[applied={T|T < 7112527580246450176}]
I20250109 21:08:48.790894 32543 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38638:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:08:48.791086 32543 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38638:
table { table_name: "test-table" } modify_external_catalogs: true
I20250109 21:08:48.793524 32543 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 1f189e97b3614de79c5bf99657c54c74: Sending DeleteTablet for 3 replicas of tablet cdb3b27087ad4d5cacaf2744cdcf6c06
I20250109 21:08:48.794368 32665 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb3b27087ad4d5cacaf2744cdcf6c06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:48 UTC) from {username='slave'} at 127.0.0.1:53850
I20250109 21:08:48.794510 328 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb3b27087ad4d5cacaf2744cdcf6c06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:48 UTC) from {username='slave'} at 127.0.0.1:57732
I20250109 21:08:48.794915 459 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb3b27087ad4d5cacaf2744cdcf6c06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 21:08:48 UTC) from {username='slave'} at 127.0.0.1:57272
I20250109 21:08:48.794991 567 tablet_replica.cc:331] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1: stopping tablet replica
I20250109 21:08:48.795274 567 raft_consensus.cc:2238] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:08:48.795341 568 tablet_replica.cc:331] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4: stopping tablet replica
I20250109 21:08:48.795660 567 raft_consensus.cc:2267] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:48.795670 568 raft_consensus.cc:2238] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 21:08:48.795905 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 32603
I20250109 21:08:48.796147 568 raft_consensus.cc:2267] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 21:08:48.797652 567 ts_tablet_manager.cc:1905] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:08:48.799433 568 ts_tablet_manager.cc:1905] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 21:08:48.800491 567 ts_tablet_manager.cc:1918] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:08:48.800581 567 log.cc:1198] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-1/wal/wals/cdb3b27087ad4d5cacaf2744cdcf6c06
I20250109 21:08:48.800923 567 ts_tablet_manager.cc:1939] T cdb3b27087ad4d5cacaf2744cdcf6c06 P c43d71d6bba449908f6df0da74ab07c1: Deleting consensus metadata
I20250109 21:08:48.801469 568 ts_tablet_manager.cc:1918] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 21:08:48.801548 568 log.cc:1198] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4: Deleting WAL directory at /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1736456573526088-12793-0/minicluster-data/ts-2/wal/wals/cdb3b27087ad4d5cacaf2744cdcf6c06
I20250109 21:08:48.801891 568 ts_tablet_manager.cc:1939] T cdb3b27087ad4d5cacaf2744cdcf6c06 P 0c1cd9e415124aefa0f02d4b11002aa4: Deleting consensus metadata
I20250109 21:08:48.802099 32527 catalog_manager.cc:4872] TS c43d71d6bba449908f6df0da74ab07c1 (127.12.126.66:43029): tablet cdb3b27087ad4d5cacaf2744cdcf6c06 (table test-table [id=80e4cc2833de4a43ac71fea3e3e904c1]) successfully deleted
I20250109 21:08:48.803100 32527 catalog_manager.cc:4872] TS 0c1cd9e415124aefa0f02d4b11002aa4 (127.12.126.67:40209): tablet cdb3b27087ad4d5cacaf2744cdcf6c06 (table test-table [id=80e4cc2833de4a43ac71fea3e3e904c1]) successfully deleted
W20250109 21:08:48.803254 32529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.126.65:34567 (error 108)
I20250109 21:08:48.803357 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 32734
I20250109 21:08:48.804519 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:48.804271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.65:34567 (local address 127.0.0.1:53866)
0109 21:08:48.804396 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0109 21:08:48.804449 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.65:34567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20250109 21:08:48.804653 32529 catalog_manager.cc:4615] TS 7dc83f9bb94140bd801a9df8940002c4 (127.12.126.65:34567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cdb3b27087ad4d5cacaf2744cdcf6c06: Network error: Client connection negotiation failed: client connection to 127.12.126.65:34567: connect: Connection refused (error 111)
I20250109 21:08:48.809602 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 397
I20250109 21:08:48.815907 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 32512
2025-01-09T21:08:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1201 ms)
[----------] 6 tests from EncryptionPolicyTest (9792 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:08:48 dist-test-slave-g0zb krb5kdc[572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:08:48 dist-test-slave-g0zb krb5kdc[572](info): set up 2 sockets
Jan 09 21:08:48 dist-test-slave-g0zb krb5kdc[572](info): commencing operation
krb5kdc: starting...
W20250109 21:08:49.855646 12793 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:08:49 dist-test-slave-g0zb krb5kdc[572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:08:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:08:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20250109 21:08:49.967679 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36227
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:50.075001 585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:50.075238 585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:50.075352 585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:50.078464 585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:08:50.078534 585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:50.078577 585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:50.078619 585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:08:50.078660 585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:08:50.082875 585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36227
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:50.084002 585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:50.084856 585 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:50.091617 590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:50.092279 593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:50.092792 591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:50.093751 585 server_base.cc:1034] running on GCE node
I20250109 21:08:50.094204 585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:50.095147 585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:50.096302 585 hybrid_clock.cc:648] HybridClock initialized: now 1736456930096278 us; error 36 us; skew 500 ppm
Jan 09 21:08:50 dist-test-slave-g0zb krb5kdc[572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:50.099071 585 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20250109 21:08:50.099937 585 webserver.cc:458] Webserver started at http://127.0.0.1:41647/ using document root <none> and password file <none>
I20250109 21:08:50.100445 585 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:50.100497 585 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:50.100642 585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:50.102397 585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "98fbb360df734243add908a97398e2a7"
format_stamp: "Formatted at 2025-01-09 21:08:50 on dist-test-slave-g0zb"
server_key: "6182ff75e915287ed5b60997fed83b17"
server_key_iv: "83edde984b2a732261631b359e5f6804"
server_key_version: "encryptionkey@0"
I20250109 21:08:50.102887 585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "98fbb360df734243add908a97398e2a7"
format_stamp: "Formatted at 2025-01-09 21:08:50 on dist-test-slave-g0zb"
server_key: "6182ff75e915287ed5b60997fed83b17"
server_key_iv: "83edde984b2a732261631b359e5f6804"
server_key_version: "encryptionkey@0"
I20250109 21:08:50.106050 585 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20250109 21:08:50.108127 600 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:50.109099 585 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250109 21:08:50.109189 585 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "98fbb360df734243add908a97398e2a7"
format_stamp: "Formatted at 2025-01-09 21:08:50 on dist-test-slave-g0zb"
server_key: "6182ff75e915287ed5b60997fed83b17"
server_key_iv: "83edde984b2a732261631b359e5f6804"
server_key_version: "encryptionkey@0"
I20250109 21:08:50.109264 585 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:50.129119 585 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:50.131667 585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:50.131814 585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:50.138139 585 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36227
I20250109 21:08:50.138154 652 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:36227 every 8 connection(s)
I20250109 21:08:50.139096 585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:08:50.141887 653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:50.143016 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 585
I20250109 21:08:50.143141 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:08:50.143388 12793 external_mini_cluster.cc:1453] Setting key 4ba8d55fc33f0254ff9c23bdd4f2113d
I20250109 21:08:50.147812 653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7: Bootstrap starting.
I20250109 21:08:50.150288 653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:50.151124 653 log.cc:826] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7: Log is configured to *not* fsync() on all Append() calls
Jan 09 21:08:50 dist-test-slave-g0zb krb5kdc[572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20250109 21:08:50.153168 653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7: No bootstrap required, opened a new log
I20250109 21:08:50.156666 656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:50.144497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34487 (local address 127.0.0.1:36227)
0109 21:08:50.144882 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:50.144895 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:50.144934 (+ 39us) server_negotiation.cc:407] Connection header received
0109 21:08:50.145664 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:50.145688 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:50.146043 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:50.146325 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:50.147413 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:50.148221 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:50.149066 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:50.149269 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:50.151940 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:50.151964 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:50.151980 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:50.152014 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:50.154030 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:50.154472 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:50.154478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:50.154483 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:50.154567 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:50.154817 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:50.154821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:50.154823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:50.155246 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:50.155452 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:50.155688 (+ 236us) server_negotiation.cc:299] Negotiation successful
0109 21:08:50.155920 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":98,"threads_started":1}
I20250109 21:08:50.156769 653 raft_consensus.cc:357] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98fbb360df734243add908a97398e2a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36227 } }
I20250109 21:08:50.156980 653 raft_consensus.cc:383] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:50.157058 653 raft_consensus.cc:738] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98fbb360df734243add908a97398e2a7, State: Initialized, Role: FOLLOWER
I20250109 21:08:50.157491 653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98fbb360df734243add908a97398e2a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36227 } }
I20250109 21:08:50.157644 653 raft_consensus.cc:397] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:08:50.157737 653 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:08:50.157860 653 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:50.158828 653 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98fbb360df734243add908a97398e2a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36227 } }
I20250109 21:08:50.159168 653 leader_election.cc:304] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98fbb360df734243add908a97398e2a7; no voters:
I20250109 21:08:50.159438 653 leader_election.cc:290] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:08:50.159526 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:08:50.159552 658 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:50.159758 658 raft_consensus.cc:695] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7 [term 1 LEADER]: Becoming Leader. State: Replica: 98fbb360df734243add908a97398e2a7, State: Running, Role: LEADER
I20250109 21:08:50.160090 658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98fbb360df734243add908a97398e2a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36227 } }
I20250109 21:08:50.160590 653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:08:50.161509 660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98fbb360df734243add908a97398e2a7. Latest consensus state: current_term: 1 leader_uuid: "98fbb360df734243add908a97398e2a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98fbb360df734243add908a97398e2a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36227 } } }
I20250109 21:08:50.161621 660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:50.161933 659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98fbb360df734243add908a97398e2a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98fbb360df734243add908a97398e2a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36227 } } }
I20250109 21:08:50.162005 659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:50.162292 671 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:08:50.165722 671 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:08:50.170990 671 catalog_manager.cc:1348] Generated new cluster ID: 5ef787672dc64947bfd6824a8189a556
I20250109 21:08:50.171097 671 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:08:50.171695 656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:50.161567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49725 (local address 127.0.0.1:36227)
0109 21:08:50.161683 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:50.161704 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:50.162816 (+ 1112us) server_negotiation.cc:407] Connection header received
0109 21:08:50.163050 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:50.163054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:50.163107 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:50.163187 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:50.163922 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:50.164580 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:50.165366 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:50.165553 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:50.166972 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:50.166990 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:50.166993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:50.167021 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:50.168844 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:50.169762 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:50.169766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:50.169767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:50.169831 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:50.170240 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:50.170245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:50.170250 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:50.170418 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:50.170556 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:50.171449 (+ 893us) server_negotiation.cc:299] Negotiation successful
0109 21:08:50.171544 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250109 21:08:50.183506 671 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:08:50.184403 671 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:08:50.193811 671 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 98fbb360df734243add908a97398e2a7: Generated new TSK 0
I20250109 21:08:50.194469 671 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:08:50.198451 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 585
W20250109 21:08:50.204208 570 connection.cc:537] client connection to 127.0.0.1:36227 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36227): Connection reset by peer (error 104)
2025-01-09T21:08:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:08:50 dist-test-slave-g0zb krb5kdc[684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:08:50 dist-test-slave-g0zb krb5kdc[684](info): set up 2 sockets
Jan 09 21:08:50 dist-test-slave-g0zb krb5kdc[684](info): commencing operation
krb5kdc: starting...
W20250109 21:08:51.227669 12793 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:08:51 dist-test-slave-g0zb krb5kdc[684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:08:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:08:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20250109 21:08:51.335867 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:51.443060 697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:51.443300 697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:51.443393 697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:51.446390 697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:08:51.446453 697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:51.446491 697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:51.446527 697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:08:51.446565 697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:08:51.450723 697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34831
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:51.451756 697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:51.452551 697 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:51.459427 703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:51.460129 705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:51.460772 702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:51.461522 697 server_base.cc:1034] running on GCE node
I20250109 21:08:51.462034 697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:51.462949 697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:51.464095 697 hybrid_clock.cc:648] HybridClock initialized: now 1736456931464078 us; error 30 us; skew 500 ppm
Jan 09 21:08:51 dist-test-slave-g0zb krb5kdc[684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:51.466656 697 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20250109 21:08:51.467510 697 webserver.cc:458] Webserver started at http://127.0.0.1:36787/ using document root <none> and password file <none>
I20250109 21:08:51.468027 697 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:51.468086 697 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:51.468271 697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:51.469986 697 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "762ef2df7ec24e3586dbc1942d35e4e2"
format_stamp: "Formatted at 2025-01-09 21:08:51 on dist-test-slave-g0zb"
server_key: "424af1b80fbce1c721ea89c4f8fc580d"
server_key_iv: "5128967a3ddf91801ed469583907985f"
server_key_version: "encryptionkey@0"
I20250109 21:08:51.470444 697 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "762ef2df7ec24e3586dbc1942d35e4e2"
format_stamp: "Formatted at 2025-01-09 21:08:51 on dist-test-slave-g0zb"
server_key: "424af1b80fbce1c721ea89c4f8fc580d"
server_key_iv: "5128967a3ddf91801ed469583907985f"
server_key_version: "encryptionkey@0"
I20250109 21:08:51.473551 697 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20250109 21:08:51.475580 712 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:51.476529 697 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:51.476639 697 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "762ef2df7ec24e3586dbc1942d35e4e2"
format_stamp: "Formatted at 2025-01-09 21:08:51 on dist-test-slave-g0zb"
server_key: "424af1b80fbce1c721ea89c4f8fc580d"
server_key_iv: "5128967a3ddf91801ed469583907985f"
server_key_version: "encryptionkey@0"
I20250109 21:08:51.476718 697 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:51.501397 697 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:51.504540 697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:51.504714 697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:51.511350 697 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34831
I20250109 21:08:51.511358 764 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:34831 every 8 connection(s)
I20250109 21:08:51.512298 697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:08:51.515213 765 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:51.520179 765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2: Bootstrap starting.
I20250109 21:08:51.521226 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 697
I20250109 21:08:51.521378 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:08:51.521654 12793 external_mini_cluster.cc:1453] Setting key 6860db922596cbed0bc0a3eed2d67227
I20250109 21:08:51.522886 765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:51.523581 765 log.cc:826] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:51.525287 765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2: No bootstrap required, opened a new log
I20250109 21:08:51.528503 765 raft_consensus.cc:357] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "762ef2df7ec24e3586dbc1942d35e4e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34831 } }
I20250109 21:08:51.528679 765 raft_consensus.cc:383] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:51.528731 765 raft_consensus.cc:738] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762ef2df7ec24e3586dbc1942d35e4e2, State: Initialized, Role: FOLLOWER
I20250109 21:08:51.528873 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:51.522676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60951 (local address 127.0.0.1:34831)
0109 21:08:51.523160 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:51.523169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:51.523203 (+ 34us) server_negotiation.cc:407] Connection header received
0109 21:08:51.523800 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:51.523819 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:51.524172 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:51.524542 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:51.524983 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:51.525865 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:51.526512 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:51.526785 (+ 273us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:51.527417 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:51.527437 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:51.527450 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:51.527798 (+ 348us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:51.527829 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:51.527847 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:51.527983 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:51.528163 (+ 180us) server_negotiation.cc:299] Negotiation successful
0109 21:08:51.528274 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":152,"threads_started":1}
I20250109 21:08:51.529204 765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "762ef2df7ec24e3586dbc1942d35e4e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34831 } }
I20250109 21:08:51.529363 765 raft_consensus.cc:397] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:08:51.529428 765 raft_consensus.cc:491] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:08:51.529538 765 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:51.530508 765 raft_consensus.cc:513] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "762ef2df7ec24e3586dbc1942d35e4e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34831 } }
I20250109 21:08:51.530874 765 leader_election.cc:304] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 762ef2df7ec24e3586dbc1942d35e4e2; no voters:
I20250109 21:08:51.531213 765 leader_election.cc:290] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:08:51.531299 770 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:51.531493 770 raft_consensus.cc:695] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2 [term 1 LEADER]: Becoming Leader. State: Replica: 762ef2df7ec24e3586dbc1942d35e4e2, State: Running, Role: LEADER
I20250109 21:08:51.531733 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:08:51.531841 770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "762ef2df7ec24e3586dbc1942d35e4e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34831 } }
I20250109 21:08:51.532352 765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:08:51.534179 772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 762ef2df7ec24e3586dbc1942d35e4e2. Latest consensus state: current_term: 1 leader_uuid: "762ef2df7ec24e3586dbc1942d35e4e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "762ef2df7ec24e3586dbc1942d35e4e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34831 } } }
I20250109 21:08:51.534296 772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:51.534686 790 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:08:51.534647 771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "762ef2df7ec24e3586dbc1942d35e4e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "762ef2df7ec24e3586dbc1942d35e4e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34831 } } }
I20250109 21:08:51.535609 771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:51.538360 790 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:08:51.539848 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:51.534506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37489 (local address 127.0.0.1:34831)
0109 21:08:51.534687 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:51.534693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:51.534707 (+ 14us) server_negotiation.cc:407] Connection header received
0109 21:08:51.534741 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:51.534743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:51.534779 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:51.534878 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:51.535303 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:51.535898 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:51.536478 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:51.536671 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:51.539376 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:51.539393 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:51.539399 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:51.539413 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:51.539441 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:51.539452 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:51.539531 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:51.539652 (+ 121us) server_negotiation.cc:299] Negotiation successful
0109 21:08:51.539684 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20250109 21:08:51.543538 790 catalog_manager.cc:1348] Generated new cluster ID: a20f2a98eea6407e9647cafbfeafd306
I20250109 21:08:51.543612 790 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:08:51.554682 790 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:08:51.555886 790 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:08:51.561872 790 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 762ef2df7ec24e3586dbc1942d35e4e2: Generated new TSK 0
I20250109 21:08:51.562655 790 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:08:51.567754 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 697
W20250109 21:08:51.573171 682 connection.cc:537] client connection to 127.0.0.1:34831 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34831): Connection reset by peer (error 104)
2025-01-09T21:08:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:08:51 dist-test-slave-g0zb krb5kdc[796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:08:51 dist-test-slave-g0zb krb5kdc[796](info): set up 2 sockets
Jan 09 21:08:51 dist-test-slave-g0zb krb5kdc[796](info): commencing operation
krb5kdc: starting...
W20250109 21:08:52.591704 12793 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:08:52 dist-test-slave-g0zb krb5kdc[796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:08:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:08:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20250109 21:08:52.703382 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:52.808013 809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:52.808247 809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:52.808320 809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:52.811420 809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:08:52.811476 809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:52.811499 809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:52.811538 809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:08:52.811573 809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:08:52.815621 809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36643
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:52.816623 809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:52.817453 809 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:52.824803 814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:52.824956 815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:52.825506 817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:52.826292 809 server_base.cc:1034] running on GCE node
I20250109 21:08:52.826731 809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:52.827611 809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:52.828770 809 hybrid_clock.cc:648] HybridClock initialized: now 1736456932828746 us; error 35 us; skew 500 ppm
Jan 09 21:08:52 dist-test-slave-g0zb krb5kdc[796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:52.831418 809 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20250109 21:08:52.832319 809 webserver.cc:458] Webserver started at http://127.0.0.1:44603/ using document root <none> and password file <none>
I20250109 21:08:52.832824 809 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:52.832875 809 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:52.833016 809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:52.834714 809 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "09acee50e2814e74a0de563c4a491b24"
format_stamp: "Formatted at 2025-01-09 21:08:52 on dist-test-slave-g0zb"
server_key: "b62f86e9d1b78f708f8d0513fd22bbaa"
server_key_iv: "7e1b9e3fdb98d02c5cc24aad235c12ea"
server_key_version: "encryptionkey@0"
I20250109 21:08:52.835224 809 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "09acee50e2814e74a0de563c4a491b24"
format_stamp: "Formatted at 2025-01-09 21:08:52 on dist-test-slave-g0zb"
server_key: "b62f86e9d1b78f708f8d0513fd22bbaa"
server_key_iv: "7e1b9e3fdb98d02c5cc24aad235c12ea"
server_key_version: "encryptionkey@0"
I20250109 21:08:52.838366 809 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250109 21:08:52.840415 824 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:52.841361 809 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:08:52.841478 809 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "09acee50e2814e74a0de563c4a491b24"
format_stamp: "Formatted at 2025-01-09 21:08:52 on dist-test-slave-g0zb"
server_key: "b62f86e9d1b78f708f8d0513fd22bbaa"
server_key_iv: "7e1b9e3fdb98d02c5cc24aad235c12ea"
server_key_version: "encryptionkey@0"
I20250109 21:08:52.841560 809 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:52.846324 809 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:52.848985 809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:52.849144 809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:52.855252 809 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36643
I20250109 21:08:52.855248 876 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:36643 every 8 connection(s)
I20250109 21:08:52.856168 809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:08:52.858438 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 809
I20250109 21:08:52.858565 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:08:52.858850 12793 external_mini_cluster.cc:1453] Setting key 9c05acc3fb9da55aa5a72f39d7089180
I20250109 21:08:52.858966 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:52.863544 880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:52.860003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43305 (local address 127.0.0.1:36643)
0109 21:08:52.860396 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:52.860407 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:52.860447 (+ 40us) server_negotiation.cc:407] Connection header received
0109 21:08:52.861075 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:52.861101 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:52.861407 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:52.861809 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:52.862411 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:52.862415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:52.862427 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:52.862440 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:52.862478 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:52.862497 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:52.862641 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:52.862715 (+ 74us) server_negotiation.cc:299] Negotiation successful
0109 21:08:52.862873 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
I20250109 21:08:52.865239 877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24: Bootstrap starting.
I20250109 21:08:52.865958 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:08:52.867417 877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:52.868137 877 log.cc:826] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:52.869378 880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:52.867802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57671 (local address 127.0.0.1:36643)
0109 21:08:52.867920 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:52.867926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:52.868100 (+ 174us) server_negotiation.cc:407] Connection header received
0109 21:08:52.868231 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:52.868234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:52.868267 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:52.868333 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:52.869017 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:52.869020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:52.869023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:52.869034 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:52.869068 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:52.869078 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:52.869152 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:52.869225 (+ 73us) server_negotiation.cc:299] Negotiation successful
0109 21:08:52.869260 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250109 21:08:52.869868 877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24: No bootstrap required, opened a new log
I20250109 21:08:52.872985 877 raft_consensus.cc:357] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09acee50e2814e74a0de563c4a491b24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36643 } }
I20250109 21:08:52.873162 877 raft_consensus.cc:383] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:52.873235 877 raft_consensus.cc:738] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09acee50e2814e74a0de563c4a491b24, State: Initialized, Role: FOLLOWER
I20250109 21:08:52.873615 877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09acee50e2814e74a0de563c4a491b24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36643 } }
I20250109 21:08:52.873740 877 raft_consensus.cc:397] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:08:52.873801 877 raft_consensus.cc:491] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:08:52.873908 877 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:52.874650 877 raft_consensus.cc:513] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09acee50e2814e74a0de563c4a491b24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36643 } }
I20250109 21:08:52.874992 877 leader_election.cc:304] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09acee50e2814e74a0de563c4a491b24; no voters:
I20250109 21:08:52.875239 877 leader_election.cc:290] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:08:52.875336 887 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:52.875523 887 raft_consensus.cc:695] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24 [term 1 LEADER]: Becoming Leader. State: Replica: 09acee50e2814e74a0de563c4a491b24, State: Running, Role: LEADER
I20250109 21:08:52.875867 887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09acee50e2814e74a0de563c4a491b24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36643 } }
I20250109 21:08:52.876333 877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:08:52.877522 889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09acee50e2814e74a0de563c4a491b24. Latest consensus state: current_term: 1 leader_uuid: "09acee50e2814e74a0de563c4a491b24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09acee50e2814e74a0de563c4a491b24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36643 } } }
I20250109 21:08:52.877650 889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:52.877596 888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09acee50e2814e74a0de563c4a491b24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09acee50e2814e74a0de563c4a491b24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36643 } } }
I20250109 21:08:52.877696 888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:52.878027 896 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:08:52.880870 896 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:08:52.885668 896 catalog_manager.cc:1348] Generated new cluster ID: b7b7a8e53ee8489e989af4ef149730a2
I20250109 21:08:52.885735 896 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:08:52.895772 896 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:08:52.896076 896 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:08:52.902436 896 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 09acee50e2814e74a0de563c4a491b24: Generated new TSK 0
I20250109 21:08:52.903228 896 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:08:52.906955 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 809
2025-01-09T21:08:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:08:52 dist-test-slave-g0zb krb5kdc[908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:08:52 dist-test-slave-g0zb krb5kdc[908](info): set up 2 sockets
Jan 09 21:08:52 dist-test-slave-g0zb krb5kdc[908](info): commencing operation
krb5kdc: starting...
W20250109 21:08:53.931506 12793 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:08:54 dist-test-slave-g0zb krb5kdc[908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:08:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:08:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20250109 21:08:54.043680 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:54.148154 921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:54.148414 921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:54.148499 921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:54.151536 921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:08:54.151595 921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:54.151633 921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:54.151675 921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:08:54.151711 921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:08:54.155578 921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:54.156528 921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:54.157311 921 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:54.164011 927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:54.164795 929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:54.165304 926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:54.166133 921 server_base.cc:1034] running on GCE node
I20250109 21:08:54.166545 921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:54.167397 921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:54.168543 921 hybrid_clock.cc:648] HybridClock initialized: now 1736456934168534 us; error 24 us; skew 500 ppm
Jan 09 21:08:54 dist-test-slave-g0zb krb5kdc[908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:54.176460 921 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20250109 21:08:54.177345 921 webserver.cc:458] Webserver started at http://127.0.0.1:44065/ using document root <none> and password file <none>
I20250109 21:08:54.177868 921 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:54.177917 921 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:54.178071 921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:54.179826 921 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "a422200aa2b74af3b98d9fa55944bdd2"
format_stamp: "Formatted at 2025-01-09 21:08:54 on dist-test-slave-g0zb"
server_key: "7d5b35ef80fb2cdc04d5bc063e00fdd0"
server_key_iv: "7e6d6faea15774711b2e37d565383bc9"
server_key_version: "encryptionkey@0"
I20250109 21:08:54.180294 921 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "a422200aa2b74af3b98d9fa55944bdd2"
format_stamp: "Formatted at 2025-01-09 21:08:54 on dist-test-slave-g0zb"
server_key: "7d5b35ef80fb2cdc04d5bc063e00fdd0"
server_key_iv: "7e6d6faea15774711b2e37d565383bc9"
server_key_version: "encryptionkey@0"
I20250109 21:08:54.183531 921 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:08:54.185478 936 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:54.186442 921 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:54.186555 921 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "a422200aa2b74af3b98d9fa55944bdd2"
format_stamp: "Formatted at 2025-01-09 21:08:54 on dist-test-slave-g0zb"
server_key: "7d5b35ef80fb2cdc04d5bc063e00fdd0"
server_key_iv: "7e6d6faea15774711b2e37d565383bc9"
server_key_version: "encryptionkey@0"
I20250109 21:08:54.186627 921 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:54.201965 921 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:54.204546 921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:54.204680 921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:54.210879 988 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:36653 every 8 connection(s)
I20250109 21:08:54.210886 921 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36653
I20250109 21:08:54.211863 921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:08:54.214838 989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:54.219058 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 921
I20250109 21:08:54.219185 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:08:54.219404 12793 external_mini_cluster.cc:1453] Setting key 57711fc5aad106f62eff962c142ad7fa
I20250109 21:08:54.219702 989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2: Bootstrap starting.
I20250109 21:08:54.221877 989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:54.222540 989 log.cc:826] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:54.224151 989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2: No bootstrap required, opened a new log
Jan 09 21:08:54 dist-test-slave-g0zb krb5kdc[908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20250109 21:08:54.227661 989 raft_consensus.cc:357] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a422200aa2b74af3b98d9fa55944bdd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36653 } }
I20250109 21:08:54.227886 989 raft_consensus.cc:383] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:54.227948 989 raft_consensus.cc:738] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a422200aa2b74af3b98d9fa55944bdd2, State: Initialized, Role: FOLLOWER
I20250109 21:08:54.228399 989 consensus_queue.cc:260] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a422200aa2b74af3b98d9fa55944bdd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36653 } }
I20250109 21:08:54.228515 989 raft_consensus.cc:397] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:08:54.228571 989 raft_consensus.cc:491] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:08:54.228704 989 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:54.229488 989 raft_consensus.cc:513] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a422200aa2b74af3b98d9fa55944bdd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36653 } }
I20250109 21:08:54.229852 989 leader_election.cc:304] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a422200aa2b74af3b98d9fa55944bdd2; no voters:
I20250109 21:08:54.230178 989 leader_election.cc:290] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:08:54.230293 994 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:54.230453 994 raft_consensus.cc:695] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2 [term 1 LEADER]: Becoming Leader. State: Replica: a422200aa2b74af3b98d9fa55944bdd2, State: Running, Role: LEADER
I20250109 21:08:54.230784 994 consensus_queue.cc:237] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a422200aa2b74af3b98d9fa55944bdd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36653 } }
I20250109 21:08:54.231338 989 sys_catalog.cc:564] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:08:54.231964 994 sys_catalog.cc:455] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a422200aa2b74af3b98d9fa55944bdd2. Latest consensus state: current_term: 1 leader_uuid: "a422200aa2b74af3b98d9fa55944bdd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a422200aa2b74af3b98d9fa55944bdd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36653 } } }
I20250109 21:08:54.232095 994 sys_catalog.cc:458] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:54.232417 992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:54.220500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41489 (local address 127.0.0.1:36653)
0109 21:08:54.220873 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:54.220882 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:54.220916 (+ 34us) server_negotiation.cc:407] Connection header received
0109 21:08:54.221450 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:54.221466 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:54.221774 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:54.222090 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:54.223044 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:54.223833 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:54.224501 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:54.224776 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:54.227242 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:54.227260 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:54.227270 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:54.227301 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:54.229272 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:54.229729 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:54.229734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:54.229738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:54.229804 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:54.230060 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:54.230063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:54.230064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:54.230446 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:54.231218 (+ 772us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:54.231353 (+ 135us) server_negotiation.cc:299] Negotiation successful
0109 21:08:54.231593 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":108,"threads_started":1}
I20250109 21:08:54.232417 995 sys_catalog.cc:455] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a422200aa2b74af3b98d9fa55944bdd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a422200aa2b74af3b98d9fa55944bdd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36653 } } }
I20250109 21:08:54.232625 998 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:08:54.233590 995 sys_catalog.cc:458] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:54.235391 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:08:54.235754 998 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:08:54.240868 998 catalog_manager.cc:1348] Generated new cluster ID: 9a7aa13467e34c008f60f5decdb3035b
I20250109 21:08:54.240926 998 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:08:54.245952 992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:54.237395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46561 (local address 127.0.0.1:36653)
0109 21:08:54.237521 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:54.237525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:54.237894 (+ 369us) server_negotiation.cc:407] Connection header received
0109 21:08:54.238099 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:54.238102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:54.238150 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:54.238200 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:54.239280 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:54.239836 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:54.240666 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:54.240885 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:54.242526 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:54.242542 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:54.242545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:54.242577 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:54.244255 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:54.244727 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:54.244731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:54.244733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:54.244782 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:54.245061 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:54.245064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:54.245067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:54.245238 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:54.245340 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:54.245678 (+ 338us) server_negotiation.cc:299] Negotiation successful
0109 21:08:54.245797 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250109 21:08:54.251466 998 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:08:54.252571 998 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:08:54.278995 998 catalog_manager.cc:5899] T 00000000000000000000000000000000 P a422200aa2b74af3b98d9fa55944bdd2: Generated new TSK 0
I20250109 21:08:54.279707 998 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:08:54.287947 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 921
2025-01-09T21:08:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:08:54 dist-test-slave-g0zb krb5kdc[1020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:08:54 dist-test-slave-g0zb krb5kdc[1020](info): set up 2 sockets
Jan 09 21:08:54 dist-test-slave-g0zb krb5kdc[1020](info): commencing operation
krb5kdc: starting...
W20250109 21:08:55.283329 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:08:55 dist-test-slave-g0zb krb5kdc[1020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:08:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:08:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20250109 21:08:55.391659 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:55.490715 1033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:55.490983 1033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:55.491073 1033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:55.494091 1033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:08:55.494150 1033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:55.494187 1033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:55.494225 1033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:08:55.494261 1033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:08:55.498241 1033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:55.499307 1033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:55.500105 1033 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:55.507328 1039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:55.507989 1038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:55.508205 1041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:55.508937 1033 server_base.cc:1034] running on GCE node
I20250109 21:08:55.509387 1033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:55.510212 1033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:55.511349 1033 hybrid_clock.cc:648] HybridClock initialized: now 1736456935511325 us; error 36 us; skew 500 ppm
Jan 09 21:08:55 dist-test-slave-g0zb krb5kdc[1020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:55.513787 1033 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20250109 21:08:55.514578 1033 webserver.cc:458] Webserver started at http://127.0.0.1:38031/ using document root <none> and password file <none>
I20250109 21:08:55.515120 1033 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:55.515179 1033 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:55.515358 1033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:55.517030 1033 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "3435c20998b144499be46e6cc74148cb"
format_stamp: "Formatted at 2025-01-09 21:08:55 on dist-test-slave-g0zb"
server_key: "d92939a13608d3bfec2c90068b16bcfe"
server_key_iv: "e1c63561d4e9500d8e3f426295ad0863"
server_key_version: "encryptionkey@0"
I20250109 21:08:55.517472 1033 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "3435c20998b144499be46e6cc74148cb"
format_stamp: "Formatted at 2025-01-09 21:08:55 on dist-test-slave-g0zb"
server_key: "d92939a13608d3bfec2c90068b16bcfe"
server_key_iv: "e1c63561d4e9500d8e3f426295ad0863"
server_key_version: "encryptionkey@0"
I20250109 21:08:55.520607 1033 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:08:55.522572 1048 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:55.523527 1033 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:55.523626 1033 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "3435c20998b144499be46e6cc74148cb"
format_stamp: "Formatted at 2025-01-09 21:08:55 on dist-test-slave-g0zb"
server_key: "d92939a13608d3bfec2c90068b16bcfe"
server_key_iv: "e1c63561d4e9500d8e3f426295ad0863"
server_key_version: "encryptionkey@0"
I20250109 21:08:55.523700 1033 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:55.538261 1033 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:55.540768 1033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:55.540917 1033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:55.546788 1033 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39643
I20250109 21:08:55.546854 1100 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:39643 every 8 connection(s)
I20250109 21:08:55.547715 1033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:08:55.550417 1101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:55.555032 1101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb: Bootstrap starting.
I20250109 21:08:55.556646 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 1033
I20250109 21:08:55.556759 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:08:55.556989 12793 external_mini_cluster.cc:1453] Setting key f303138b1c22f995c606ba2ca13c96d4
I20250109 21:08:55.557145 1101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:55.557946 1101 log.cc:826] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:55.559810 1101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb: No bootstrap required, opened a new log
I20250109 21:08:55.563402 1101 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3435c20998b144499be46e6cc74148cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39643 } }
I20250109 21:08:55.563606 1101 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:55.563678 1101 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3435c20998b144499be46e6cc74148cb, State: Initialized, Role: FOLLOWER
I20250109 21:08:55.564101 1101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3435c20998b144499be46e6cc74148cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39643 } }
I20250109 21:08:55.564208 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:55.557882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52959 (local address 127.0.0.1:39643)
0109 21:08:55.558303 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:55.558316 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:55.558347 (+ 31us) server_negotiation.cc:407] Connection header received
0109 21:08:55.559064 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:55.559087 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:55.559414 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:55.559783 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:55.560186 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:55.561360 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:55.561994 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:55.562332 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:55.562620 (+ 288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:55.562638 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:55.562651 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:55.563127 (+ 476us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:55.563165 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:55.563184 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:55.563289 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:55.563383 (+ 94us) server_negotiation.cc:299] Negotiation successful
0109 21:08:55.563537 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":128,"threads_started":1}
I20250109 21:08:55.564244 1101 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:08:55.564294 1101 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:08:55.564409 1101 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:55.565172 1101 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3435c20998b144499be46e6cc74148cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39643 } }
I20250109 21:08:55.565493 1101 leader_election.cc:304] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3435c20998b144499be46e6cc74148cb; no voters:
I20250109 21:08:55.565769 1101 leader_election.cc:290] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:08:55.565853 1106 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:55.566052 1106 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb [term 1 LEADER]: Becoming Leader. State: Replica: 3435c20998b144499be46e6cc74148cb, State: Running, Role: LEADER
I20250109 21:08:55.566326 1106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3435c20998b144499be46e6cc74148cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39643 } }
I20250109 21:08:55.566711 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:08:55.566924 1101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:08:55.568902 1107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3435c20998b144499be46e6cc74148cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3435c20998b144499be46e6cc74148cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39643 } } }
I20250109 21:08:55.569031 1107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:55.569285 1108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3435c20998b144499be46e6cc74148cb. Latest consensus state: current_term: 1 leader_uuid: "3435c20998b144499be46e6cc74148cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3435c20998b144499be46e6cc74148cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39643 } } }
I20250109 21:08:55.569531 1108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb [sys.catalog]: This master's current role is: LEADER
W20250109 21:08:55.571781 1125 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:08:55.571880 1125 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 21:08:55.571995 1127 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:08:55.572957 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:55.569193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43855 (local address 127.0.0.1:39643)
0109 21:08:55.569300 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:55.569305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:55.569636 (+ 331us) server_negotiation.cc:407] Connection header received
0109 21:08:55.569837 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:55.569840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:55.569888 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:55.569981 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:55.570352 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:55.571069 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:55.571876 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:55.572096 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:55.572526 (+ 430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:55.572542 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:55.572545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:55.572557 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:55.572584 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:55.572596 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:55.572673 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:55.572776 (+ 103us) server_negotiation.cc:299] Negotiation successful
0109 21:08:55.572840 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250109 21:08:55.573276 1127 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:08:55.578066 1127 catalog_manager.cc:1348] Generated new cluster ID: 4b4d9a99d25c4f0f85c9e556c48c4919
I20250109 21:08:55.578130 1127 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:08:55.593549 1127 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:08:55.594489 1127 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:08:55.601887 1127 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 3435c20998b144499be46e6cc74148cb: Generated new TSK 0
I20250109 21:08:55.602587 1127 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:08:55.606518 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 1033
2025-01-09T21:08:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:08:55 dist-test-slave-g0zb krb5kdc[1132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:08:55 dist-test-slave-g0zb krb5kdc[1132](info): set up 2 sockets
Jan 09 21:08:55 dist-test-slave-g0zb krb5kdc[1132](info): commencing operation
krb5kdc: starting...
W20250109 21:08:56.607429 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:08:56 dist-test-slave-g0zb krb5kdc[1132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:08:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:08:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20250109 21:08:56.710057 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:56.806998 1145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:56.807209 1145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:56.807260 1145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:56.810163 1145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:08:56.810206 1145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:56.810226 1145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:56.810242 1145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:08:56.810257 1145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:08:56.814019 1145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:56.814898 1145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:56.815613 1145 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:56.821854 1150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:56.822832 1145 server_base.cc:1034] running on GCE node
W20250109 21:08:56.823329 1153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:56.823843 1151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:56.824298 1145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:56.825088 1145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:56.826228 1145 hybrid_clock.cc:648] HybridClock initialized: now 1736456936826213 us; error 33 us; skew 500 ppm
Jan 09 21:08:56 dist-test-slave-g0zb krb5kdc[1132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:56.828732 1145 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20250109 21:08:56.829564 1145 webserver.cc:458] Webserver started at http://127.0.0.1:38727/ using document root <none> and password file <none>
I20250109 21:08:56.830044 1145 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:56.830091 1145 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:56.830226 1145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:56.831844 1145 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "a2bda264704545289ec2ab8e3989cd25"
format_stamp: "Formatted at 2025-01-09 21:08:56 on dist-test-slave-g0zb"
server_key: "2ee32d4d9fed3562f885db994ea7fcf9"
server_key_iv: "1977578fdbf8d8c912eba3e944c606ea"
server_key_version: "encryptionkey@0"
I20250109 21:08:56.832264 1145 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "a2bda264704545289ec2ab8e3989cd25"
format_stamp: "Formatted at 2025-01-09 21:08:56 on dist-test-slave-g0zb"
server_key: "2ee32d4d9fed3562f885db994ea7fcf9"
server_key_iv: "1977578fdbf8d8c912eba3e944c606ea"
server_key_version: "encryptionkey@0"
I20250109 21:08:56.835281 1145 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:08:56.837062 1160 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:56.838001 1145 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:56.838091 1145 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "a2bda264704545289ec2ab8e3989cd25"
format_stamp: "Formatted at 2025-01-09 21:08:56 on dist-test-slave-g0zb"
server_key: "2ee32d4d9fed3562f885db994ea7fcf9"
server_key_iv: "1977578fdbf8d8c912eba3e944c606ea"
server_key_version: "encryptionkey@0"
I20250109 21:08:56.838160 1145 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:56.842509 1145 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:56.844787 1145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:56.844902 1145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:56.851003 1145 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32987
I20250109 21:08:56.851024 1212 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:32987 every 8 connection(s)
I20250109 21:08:56.851794 1145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:08:56.854297 1213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:56.854540 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 1145
I20250109 21:08:56.854648 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:08:56.854944 12793 external_mini_cluster.cc:1453] Setting key 04c90767b5c71f48d2aff1b3648dd6d3
I20250109 21:08:56.859083 1216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:56.855950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59747 (local address 127.0.0.1:32987)
0109 21:08:56.856305 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:56.856312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:56.856341 (+ 29us) server_negotiation.cc:407] Connection header received
0109 21:08:56.856855 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:56.856870 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:56.857081 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:56.857333 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:56.858140 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:56.858144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:56.858152 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:56.858161 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:56.858194 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:56.858208 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:56.858320 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:56.858414 (+ 94us) server_negotiation.cc:299] Negotiation successful
0109 21:08:56.858526 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":98,"threads_started":1}
I20250109 21:08:56.859689 1213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25: Bootstrap starting.
I20250109 21:08:56.861325 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:08:56.862368 1213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:56.863202 1213 log.cc:826] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:56.864856 1216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:56.863673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45745 (local address 127.0.0.1:32987)
0109 21:08:56.863798 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:56.863802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:56.863815 (+ 13us) server_negotiation.cc:407] Connection header received
0109 21:08:56.863848 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:56.863850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:56.863877 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:56.863935 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:56.864527 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:56.864529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:56.864531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:56.864540 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:56.864561 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:56.864570 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:56.864649 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:56.864742 (+ 93us) server_negotiation.cc:299] Negotiation successful
0109 21:08:56.864766 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250109 21:08:56.864970 1213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25: No bootstrap required, opened a new log
I20250109 21:08:56.867874 1213 raft_consensus.cc:357] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bda264704545289ec2ab8e3989cd25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32987 } }
I20250109 21:08:56.868032 1213 raft_consensus.cc:383] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:56.868098 1213 raft_consensus.cc:738] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2bda264704545289ec2ab8e3989cd25, State: Initialized, Role: FOLLOWER
I20250109 21:08:56.868468 1213 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bda264704545289ec2ab8e3989cd25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32987 } }
I20250109 21:08:56.868592 1213 raft_consensus.cc:397] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:08:56.868665 1213 raft_consensus.cc:491] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:08:56.868777 1213 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:56.869475 1213 raft_consensus.cc:513] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bda264704545289ec2ab8e3989cd25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32987 } }
I20250109 21:08:56.869773 1213 leader_election.cc:304] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2bda264704545289ec2ab8e3989cd25; no voters:
I20250109 21:08:56.870023 1213 leader_election.cc:290] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:08:56.870132 1223 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:56.870318 1223 raft_consensus.cc:695] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25 [term 1 LEADER]: Becoming Leader. State: Replica: a2bda264704545289ec2ab8e3989cd25, State: Running, Role: LEADER
I20250109 21:08:56.870605 1223 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bda264704545289ec2ab8e3989cd25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32987 } }
I20250109 21:08:56.870954 1213 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:08:56.871961 1224 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2bda264704545289ec2ab8e3989cd25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bda264704545289ec2ab8e3989cd25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32987 } } }
I20250109 21:08:56.872071 1225 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2bda264704545289ec2ab8e3989cd25. Latest consensus state: current_term: 1 leader_uuid: "a2bda264704545289ec2ab8e3989cd25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bda264704545289ec2ab8e3989cd25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32987 } } }
I20250109 21:08:56.872082 1224 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:56.872203 1225 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:56.872517 1232 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:08:56.874876 1232 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:08:56.879457 1232 catalog_manager.cc:1348] Generated new cluster ID: a511f05ebe364985b523567fa9968dad
I20250109 21:08:56.879516 1232 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:08:56.907892 1232 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:08:56.908196 1232 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:08:56.915685 1232 catalog_manager.cc:5899] T 00000000000000000000000000000000 P a2bda264704545289ec2ab8e3989cd25: Generated new TSK 0
I20250109 21:08:56.916414 1232 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:08:56.926242 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 1145
2025-01-09T21:08:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:08:56 dist-test-slave-g0zb krb5kdc[1244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:08:56 dist-test-slave-g0zb krb5kdc[1244](info): set up 2 sockets
Jan 09 21:08:56 dist-test-slave-g0zb krb5kdc[1244](info): commencing operation
krb5kdc: starting...
W20250109 21:08:57.907524 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:08:57 dist-test-slave-g0zb krb5kdc[1244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:08:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:08:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20250109 21:08:58.013351 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43187
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:58.111300 1257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:58.111529 1257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:58.111578 1257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:58.114354 1257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:08:58.114395 1257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:58.114413 1257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:58.114428 1257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:08:58.114442 1257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:08:58.118011 1257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:58.118840 1257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:58.119578 1257 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:58.125769 1262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:58.126431 1265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:58.127115 1263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:58.127672 1257 server_base.cc:1034] running on GCE node
I20250109 21:08:58.128091 1257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:58.128932 1257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:58.130096 1257 hybrid_clock.cc:648] HybridClock initialized: now 1736456938130082 us; error 28 us; skew 500 ppm
Jan 09 21:08:58 dist-test-slave-g0zb krb5kdc[1244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:58.132440 1257 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20250109 21:08:58.133201 1257 webserver.cc:458] Webserver started at http://127.0.0.1:42977/ using document root <none> and password file <none>
I20250109 21:08:58.133687 1257 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:58.133744 1257 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:58.133901 1257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:58.135468 1257 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "1104d89ed2d645a98c8c52f0df43016f"
format_stamp: "Formatted at 2025-01-09 21:08:58 on dist-test-slave-g0zb"
server_key: "7f62f0ba22b6003d41af67f1d80ba32c"
server_key_iv: "855c7802afc7bbf68fd9e0846ae81353"
server_key_version: "encryptionkey@0"
I20250109 21:08:58.135919 1257 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "1104d89ed2d645a98c8c52f0df43016f"
format_stamp: "Formatted at 2025-01-09 21:08:58 on dist-test-slave-g0zb"
server_key: "7f62f0ba22b6003d41af67f1d80ba32c"
server_key_iv: "855c7802afc7bbf68fd9e0846ae81353"
server_key_version: "encryptionkey@0"
I20250109 21:08:58.138995 1257 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:08:58.140969 1272 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:58.141865 1257 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:08:58.141963 1257 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "1104d89ed2d645a98c8c52f0df43016f"
format_stamp: "Formatted at 2025-01-09 21:08:58 on dist-test-slave-g0zb"
server_key: "7f62f0ba22b6003d41af67f1d80ba32c"
server_key_iv: "855c7802afc7bbf68fd9e0846ae81353"
server_key_version: "encryptionkey@0"
I20250109 21:08:58.142042 1257 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:58.165092 1257 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:58.167378 1257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:58.167520 1257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:58.173135 1257 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43187
I20250109 21:08:58.173149 1324 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:43187 every 8 connection(s)
I20250109 21:08:58.173987 1257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:08:58.176683 1325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:58.178077 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 1257
I20250109 21:08:58.178194 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:08:58.178414 12793 external_mini_cluster.cc:1453] Setting key 5548da90089c2a176b854ddbf2218906
I20250109 21:08:58.182183 1325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f: Bootstrap starting.
I20250109 21:08:58.184387 1325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:58.185016 1325 log.cc:826] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f: Log is configured to *not* fsync() on all Append() calls
Jan 09 21:08:58 dist-test-slave-g0zb krb5kdc[1244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20250109 21:08:58.186688 1325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f: No bootstrap required, opened a new log
I20250109 21:08:58.189996 1328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:58.179513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:58939 (local address 127.0.0.1:43187)
0109 21:08:58.179910 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:58.179921 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:58.179957 (+ 36us) server_negotiation.cc:407] Connection header received
0109 21:08:58.180589 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:58.180612 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:58.180950 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:58.181277 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:58.182012 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:58.183135 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:58.183717 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:58.183954 (+ 237us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:58.185781 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:58.185796 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:58.185806 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:58.185838 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:58.187621 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:58.188022 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:58.188027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:58.188032 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:58.188109 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:58.188324 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:58.188327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:58.188329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:58.188686 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:58.188910 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:58.189071 (+ 161us) server_negotiation.cc:299] Negotiation successful
0109 21:08:58.189271 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":109,"threads_started":1}
I20250109 21:08:58.190034 1325 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1104d89ed2d645a98c8c52f0df43016f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43187 } }
I20250109 21:08:58.190207 1325 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:58.190260 1325 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1104d89ed2d645a98c8c52f0df43016f, State: Initialized, Role: FOLLOWER
I20250109 21:08:58.190646 1325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1104d89ed2d645a98c8c52f0df43016f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43187 } }
I20250109 21:08:58.190778 1325 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:08:58.190855 1325 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:08:58.190943 1325 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:58.191685 1325 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1104d89ed2d645a98c8c52f0df43016f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43187 } }
I20250109 21:08:58.192004 1325 leader_election.cc:304] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1104d89ed2d645a98c8c52f0df43016f; no voters:
I20250109 21:08:58.192262 1325 leader_election.cc:290] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:08:58.192420 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:08:58.192431 1330 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:58.192572 1330 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f [term 1 LEADER]: Becoming Leader. State: Replica: 1104d89ed2d645a98c8c52f0df43016f, State: Running, Role: LEADER
I20250109 21:08:58.192914 1330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1104d89ed2d645a98c8c52f0df43016f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43187 } }
I20250109 21:08:58.193326 1325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:08:58.194032 1330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1104d89ed2d645a98c8c52f0df43016f. Latest consensus state: current_term: 1 leader_uuid: "1104d89ed2d645a98c8c52f0df43016f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1104d89ed2d645a98c8c52f0df43016f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43187 } } }
I20250109 21:08:58.194216 1330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:58.194345 1332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1104d89ed2d645a98c8c52f0df43016f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1104d89ed2d645a98c8c52f0df43016f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43187 } } }
I20250109 21:08:58.194437 1332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:58.194573 1343 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:08:58.197340 1343 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:08:58.202176 1328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:58.194935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:34815 (local address 127.0.0.1:43187)
0109 21:08:58.195042 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:58.195047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:58.195070 (+ 23us) server_negotiation.cc:407] Connection header received
0109 21:08:58.195100 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:58.195103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:58.195143 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:58.195203 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:58.196120 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:58.196720 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:58.197482 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:58.197666 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:58.199146 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:58.199160 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:58.199163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:08:58.199188 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:58.200797 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:58.201182 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:58.201185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:58.201187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:58.201234 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:08:58.201440 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:08:58.201443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:08:58.201444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:08:58.201591 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:58.201779 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:58.201949 (+ 170us) server_negotiation.cc:299] Negotiation successful
0109 21:08:58.202036 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250109 21:08:58.203928 1343 catalog_manager.cc:1348] Generated new cluster ID: 09fa72cd13a04b92b0c4c44bb0724aeb
I20250109 21:08:58.203990 1343 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:08:58.229785 1343 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:08:58.230618 1343 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:08:58.234529 1343 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 1104d89ed2d645a98c8c52f0df43016f: Generated new TSK 0
I20250109 21:08:58.235119 1343 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:08:58.242868 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 1257
W20250109 21:08:58.248121 1242 connection.cc:537] client connection to 127.0.0.1:43187 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43187): Connection reset by peer (error 104)
2025-01-09T21:08:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:08:58 dist-test-slave-g0zb krb5kdc[1356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:08:58 dist-test-slave-g0zb krb5kdc[1356](info): set up 2 sockets
Jan 09 21:08:58 dist-test-slave-g0zb krb5kdc[1356](info): commencing operation
krb5kdc: starting...
W20250109 21:08:59.215430 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:08:59 dist-test-slave-g0zb krb5kdc[1356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:08:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:08:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20250109 21:08:59.322980 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38287
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:08:59.421329 1369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:08:59.421516 1369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:08:59.421562 1369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:08:59.424676 1369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:08:59.424726 1369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:08:59.424748 1369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:08:59.424765 1369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:08:59.424782 1369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:08:59.428470 1369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:08:59.429351 1369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:08:59.430068 1369 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:08:59.436457 1377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:59.437170 1375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:08:59.437767 1374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:08:59.438567 1369 server_base.cc:1034] running on GCE node
I20250109 21:08:59.439002 1369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:08:59.439774 1369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:08:59.440919 1369 hybrid_clock.cc:648] HybridClock initialized: now 1736456939440909 us; error 31 us; skew 500 ppm
Jan 09 21:08:59 dist-test-slave-g0zb krb5kdc[1356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:08:59.447626 1369 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20250109 21:08:59.448419 1369 webserver.cc:458] Webserver started at http://127.0.0.1:41507/ using document root <none> and password file <none>
I20250109 21:08:59.448940 1369 fs_manager.cc:362] Metadata directory not provided
I20250109 21:08:59.448997 1369 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:08:59.449160 1369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:08:59.450791 1369 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "0df17c8e559242b38424fc4f2d328f41"
format_stamp: "Formatted at 2025-01-09 21:08:59 on dist-test-slave-g0zb"
server_key: "7563d51b60f9545e45685d15c752a8e1"
server_key_iv: "e5d971b2c2485984905f7f1e3e06e9c4"
server_key_version: "encryptionkey@0"
I20250109 21:08:59.451303 1369 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "0df17c8e559242b38424fc4f2d328f41"
format_stamp: "Formatted at 2025-01-09 21:08:59 on dist-test-slave-g0zb"
server_key: "7563d51b60f9545e45685d15c752a8e1"
server_key_iv: "e5d971b2c2485984905f7f1e3e06e9c4"
server_key_version: "encryptionkey@0"
I20250109 21:08:59.454409 1369 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20250109 21:08:59.456367 1384 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:08:59.457247 1369 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250109 21:08:59.457337 1369 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "0df17c8e559242b38424fc4f2d328f41"
format_stamp: "Formatted at 2025-01-09 21:08:59 on dist-test-slave-g0zb"
server_key: "7563d51b60f9545e45685d15c752a8e1"
server_key_iv: "e5d971b2c2485984905f7f1e3e06e9c4"
server_key_version: "encryptionkey@0"
I20250109 21:08:59.457425 1369 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:08:59.475610 1369 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:08:59.478055 1369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:08:59.478201 1369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:08:59.484248 1369 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38287
I20250109 21:08:59.484261 1436 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:38287 every 8 connection(s)
I20250109 21:08:59.485119 1369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:08:59.487838 1437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:08:59.488092 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 1369
I20250109 21:08:59.488214 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:08:59.488493 12793 external_mini_cluster.cc:1453] Setting key 5f49ff314ad37e746f42773fed7882cb
I20250109 21:08:59.493983 1437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41: Bootstrap starting.
I20250109 21:08:59.496184 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:59.489578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:59921 (local address 127.0.0.1:38287)
0109 21:08:59.489959 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:59.489971 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:59.490094 (+ 123us) server_negotiation.cc:407] Connection header received
0109 21:08:59.490781 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:59.490823 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:59.491215 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:59.491574 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:59.492049 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:59.493213 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:59.493936 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:59.494198 (+ 262us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:59.494618 (+ 420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:59.494679 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:59.494691 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:59.495146 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:59.495182 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:59.495203 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:59.495349 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:59.495444 (+ 95us) server_negotiation.cc:299] Negotiation successful
0109 21:08:59.495545 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":104,"threads_started":1}
I20250109 21:08:59.496675 1437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41: Neither blocks nor log segments found. Creating new log.
I20250109 21:08:59.497437 1437 log.cc:826] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41: Log is configured to *not* fsync() on all Append() calls
I20250109 21:08:59.498762 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:08:59.499327 1437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41: No bootstrap required, opened a new log
I20250109 21:08:59.502877 1437 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df17c8e559242b38424fc4f2d328f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38287 } }
I20250109 21:08:59.503116 1437 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:08:59.503448 1437 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df17c8e559242b38424fc4f2d328f41, State: Initialized, Role: FOLLOWER
I20250109 21:08:59.503881 1437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df17c8e559242b38424fc4f2d328f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38287 } }
I20250109 21:08:59.504071 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:08:59.500830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:35561 (local address 127.0.0.1:38287)
0109 21:08:59.500930 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0109 21:08:59.500934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:08:59.501156 (+ 222us) server_negotiation.cc:407] Connection header received
0109 21:08:59.501257 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:08:59.501259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:08:59.501295 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:08:59.501357 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:08:59.501801 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:59.502228 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:08:59.502964 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:08:59.503076 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:08:59.503643 (+ 567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:08:59.503660 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:08:59.503663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:08:59.503675 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:08:59.503699 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:08:59.503710 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:08:59.503788 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0109 21:08:59.503905 (+ 117us) server_negotiation.cc:299] Negotiation successful
0109 21:08:59.503930 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20250109 21:08:59.504101 1437 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:08:59.504165 1437 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:08:59.504271 1437 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:08:59.505121 1437 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df17c8e559242b38424fc4f2d328f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38287 } }
I20250109 21:08:59.505436 1437 leader_election.cc:304] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0df17c8e559242b38424fc4f2d328f41; no voters:
I20250109 21:08:59.505654 1437 leader_election.cc:290] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:08:59.505743 1447 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:08:59.505937 1447 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41 [term 1 LEADER]: Becoming Leader. State: Replica: 0df17c8e559242b38424fc4f2d328f41, State: Running, Role: LEADER
I20250109 21:08:59.506255 1447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df17c8e559242b38424fc4f2d328f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38287 } }
I20250109 21:08:59.506557 1437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:08:59.507771 1449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0df17c8e559242b38424fc4f2d328f41. Latest consensus state: current_term: 1 leader_uuid: "0df17c8e559242b38424fc4f2d328f41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df17c8e559242b38424fc4f2d328f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38287 } } }
I20250109 21:08:59.507771 1448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0df17c8e559242b38424fc4f2d328f41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df17c8e559242b38424fc4f2d328f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38287 } } }
I20250109 21:08:59.507911 1449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:59.507920 1448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41 [sys.catalog]: This master's current role is: LEADER
I20250109 21:08:59.508374 1458 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:08:59.511449 1458 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:08:59.516119 1458 catalog_manager.cc:1348] Generated new cluster ID: 630bfbd6993843d99c742024809bdb93
I20250109 21:08:59.516180 1458 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:08:59.525979 1458 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:08:59.527027 1458 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:08:59.534603 1458 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 0df17c8e559242b38424fc4f2d328f41: Generated new TSK 0
I20250109 21:08:59.535316 1458 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:08:59.539726 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 1369
W20250109 21:08:59.545421 1354 connection.cc:537] client connection to 127.0.0.1:38287 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38287): Connection reset by peer (error 104)
2025-01-09T21:08:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:08:59 dist-test-slave-g0zb krb5kdc[1468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:08:59 dist-test-slave-g0zb krb5kdc[1468](info): set up 2 sockets
Jan 09 21:08:59 dist-test-slave-g0zb krb5kdc[1468](info): commencing operation
krb5kdc: starting...
W20250109 21:09:00.535853 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:09:00 dist-test-slave-g0zb krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:09:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:09:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20250109 21:09:00.645097 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41727
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:09:00.742053 1481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:09:00.742278 1481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:09:00.742365 1481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:09:00.745309 1481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:09:00.745401 1481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:09:00.745441 1481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:09:00.745512 1481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:09:00.745553 1481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:09:00.749358 1481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:09:00.750280 1481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:09:00.751055 1481 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:09:00.757755 1489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:00.757817 1486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:00.758018 1487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:09:00.758311 1481 server_base.cc:1034] running on GCE node
I20250109 21:09:00.758730 1481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:09:00.759629 1481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:09:00.760792 1481 hybrid_clock.cc:648] HybridClock initialized: now 1736456940760766 us; error 40 us; skew 500 ppm
Jan 09 21:09:00 dist-test-slave-g0zb krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:09:00.763422 1481 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20250109 21:09:00.764256 1481 webserver.cc:458] Webserver started at http://127.0.0.1:37711/ using document root <none> and password file <none>
I20250109 21:09:00.764765 1481 fs_manager.cc:362] Metadata directory not provided
I20250109 21:09:00.764823 1481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:09:00.764981 1481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:09:00.766680 1481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "e3050126fe1f488fb2e24a4ea16a89a9"
format_stamp: "Formatted at 2025-01-09 21:09:00 on dist-test-slave-g0zb"
server_key: "113e41aac81fae8c1803e7ea8c66455f"
server_key_iv: "c842f166905c7359054dfb10d9fa1710"
server_key_version: "encryptionkey@0"
I20250109 21:09:00.767194 1481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "e3050126fe1f488fb2e24a4ea16a89a9"
format_stamp: "Formatted at 2025-01-09 21:09:00 on dist-test-slave-g0zb"
server_key: "113e41aac81fae8c1803e7ea8c66455f"
server_key_iv: "c842f166905c7359054dfb10d9fa1710"
server_key_version: "encryptionkey@0"
I20250109 21:09:00.770337 1481 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250109 21:09:00.772353 1496 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:09:00.773236 1481 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:09:00.773335 1481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "e3050126fe1f488fb2e24a4ea16a89a9"
format_stamp: "Formatted at 2025-01-09 21:09:00 on dist-test-slave-g0zb"
server_key: "113e41aac81fae8c1803e7ea8c66455f"
server_key_iv: "c842f166905c7359054dfb10d9fa1710"
server_key_version: "encryptionkey@0"
I20250109 21:09:00.773409 1481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:09:00.777794 1481 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:09:00.780395 1481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:09:00.780530 1481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:09:00.786563 1481 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41727
I20250109 21:09:00.786584 1548 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:41727 every 8 connection(s)
I20250109 21:09:00.787499 1481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:09:00.789898 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 1481
I20250109 21:09:00.790026 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:09:00.790247 1549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:09:00.790285 12793 external_mini_cluster.cc:1453] Setting key 3b146b80e23584a63229cdc0a64c6f75
I20250109 21:09:00.795284 1552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:00.791452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:51355 (local address 127.0.0.1:41727)
0109 21:09:00.791884 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:00.791894 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:00.791927 (+ 33us) server_negotiation.cc:407] Connection header received
0109 21:09:00.792669 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:00.792688 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:00.792936 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:00.793262 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:00.794049 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:00.794053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:00.794063 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:09:00.794073 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:00.794111 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:09:00.794126 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:00.794286 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:00.794365 (+ 79us) server_negotiation.cc:299] Negotiation successful
0109 21:09:00.794496 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":156,"threads_started":1}
I20250109 21:09:00.796538 1549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9: Bootstrap starting.
I20250109 21:09:00.798084 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:09:00.798836 1549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9: Neither blocks nor log segments found. Creating new log.
I20250109 21:09:00.799607 1549 log.cc:826] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9: Log is configured to *not* fsync() on all Append() calls
I20250109 21:09:00.801420 1549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9: No bootstrap required, opened a new log
I20250109 21:09:00.801589 1552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:00.800264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:44109 (local address 127.0.0.1:41727)
0109 21:09:00.800415 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:00.800420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:00.800434 (+ 14us) server_negotiation.cc:407] Connection header received
0109 21:09:00.800469 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:00.800471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:00.800499 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:00.800578 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:00.801250 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:00.801253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:00.801256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:09:00.801267 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:00.801294 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:09:00.801306 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:00.801396 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:00.801446 (+ 50us) server_negotiation.cc:299] Negotiation successful
0109 21:09:00.801484 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250109 21:09:00.804587 1549 raft_consensus.cc:357] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3050126fe1f488fb2e24a4ea16a89a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41727 } }
I20250109 21:09:00.804780 1549 raft_consensus.cc:383] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:09:00.804821 1549 raft_consensus.cc:738] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3050126fe1f488fb2e24a4ea16a89a9, State: Initialized, Role: FOLLOWER
I20250109 21:09:00.805160 1549 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3050126fe1f488fb2e24a4ea16a89a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41727 } }
I20250109 21:09:00.805259 1549 raft_consensus.cc:397] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:09:00.805301 1549 raft_consensus.cc:491] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:09:00.805379 1549 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:09:00.806239 1549 raft_consensus.cc:513] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3050126fe1f488fb2e24a4ea16a89a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41727 } }
I20250109 21:09:00.806537 1549 leader_election.cc:304] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3050126fe1f488fb2e24a4ea16a89a9; no voters:
I20250109 21:09:00.806768 1549 leader_election.cc:290] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:09:00.806906 1559 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:09:00.807101 1559 raft_consensus.cc:695] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9 [term 1 LEADER]: Becoming Leader. State: Replica: e3050126fe1f488fb2e24a4ea16a89a9, State: Running, Role: LEADER
I20250109 21:09:00.807389 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3050126fe1f488fb2e24a4ea16a89a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41727 } }
I20250109 21:09:00.807863 1549 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:09:00.808868 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3050126fe1f488fb2e24a4ea16a89a9. Latest consensus state: current_term: 1 leader_uuid: "e3050126fe1f488fb2e24a4ea16a89a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3050126fe1f488fb2e24a4ea16a89a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41727 } } }
I20250109 21:09:00.809005 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9 [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:00.809260 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3050126fe1f488fb2e24a4ea16a89a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3050126fe1f488fb2e24a4ea16a89a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41727 } } }
I20250109 21:09:00.809345 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9 [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:00.809465 1568 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:09:00.812418 1568 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:09:00.817786 1568 catalog_manager.cc:1348] Generated new cluster ID: f8b675e9457343e4b6a281e9275995a1
I20250109 21:09:00.817844 1568 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:09:00.825536 1568 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:09:00.825815 1568 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:09:00.838728 1568 catalog_manager.cc:5899] T 00000000000000000000000000000000 P e3050126fe1f488fb2e24a4ea16a89a9: Generated new TSK 0
I20250109 21:09:00.839506 1568 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:09:00.843899 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 1481
2025-01-09T21:09:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:09:00 dist-test-slave-g0zb krb5kdc[1580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:09:00 dist-test-slave-g0zb krb5kdc[1580](info): set up 2 sockets
Jan 09 21:09:00 dist-test-slave-g0zb krb5kdc[1580](info): commencing operation
krb5kdc: starting...
W20250109 21:09:01.839604 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:09:01 dist-test-slave-g0zb krb5kdc[1580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:09:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:09:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20250109 21:09:01.948197 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35311
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:09:02.051213 1593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:09:02.051455 1593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:09:02.051507 1593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:09:02.054446 1593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:09:02.054498 1593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:09:02.054520 1593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:09:02.054538 1593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:09:02.054553 1593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:09:02.058315 1593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35311
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:09:02.059347 1593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:09:02.060158 1593 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:09:02.066932 1599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:02.067138 1601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:02.067334 1598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:09:02.067569 1593 server_base.cc:1034] running on GCE node
I20250109 21:09:02.068025 1593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:09:02.068872 1593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:09:02.070019 1593 hybrid_clock.cc:648] HybridClock initialized: now 1736456942070005 us; error 29 us; skew 500 ppm
Jan 09 21:09:02 dist-test-slave-g0zb krb5kdc[1580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:09:02.072721 1593 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20250109 21:09:02.073614 1593 webserver.cc:458] Webserver started at http://127.0.0.1:40719/ using document root <none> and password file <none>
I20250109 21:09:02.074131 1593 fs_manager.cc:362] Metadata directory not provided
I20250109 21:09:02.074182 1593 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:09:02.074332 1593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:09:02.076066 1593 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "0a46131e086d47f894d1a951cf35f9b0"
format_stamp: "Formatted at 2025-01-09 21:09:02 on dist-test-slave-g0zb"
server_key: "3b7283b2159356b2cc72854b3000ba3c"
server_key_iv: "95244da5c30e4a0af37f47ea651dba60"
server_key_version: "encryptionkey@0"
I20250109 21:09:02.076499 1593 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "0a46131e086d47f894d1a951cf35f9b0"
format_stamp: "Formatted at 2025-01-09 21:09:02 on dist-test-slave-g0zb"
server_key: "3b7283b2159356b2cc72854b3000ba3c"
server_key_iv: "95244da5c30e4a0af37f47ea651dba60"
server_key_version: "encryptionkey@0"
I20250109 21:09:02.079723 1593 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:09:02.081780 1608 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:09:02.082701 1593 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 21:09:02.082832 1593 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "0a46131e086d47f894d1a951cf35f9b0"
format_stamp: "Formatted at 2025-01-09 21:09:02 on dist-test-slave-g0zb"
server_key: "3b7283b2159356b2cc72854b3000ba3c"
server_key_iv: "95244da5c30e4a0af37f47ea651dba60"
server_key_version: "encryptionkey@0"
I20250109 21:09:02.082921 1593 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:09:02.104301 1593 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:09:02.107086 1593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:09:02.107263 1593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:09:02.113742 1593 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35311
I20250109 21:09:02.113734 1661 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:35311 every 8 connection(s)
I20250109 21:09:02.114728 1593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:09:02.117542 1662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:09:02.122435 1662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0: Bootstrap starting.
I20250109 21:09:02.123975 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 1593
I20250109 21:09:02.124119 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:09:02.124382 12793 external_mini_cluster.cc:1453] Setting key 1158a9983fb97c98e658af611a2a9016
I20250109 21:09:02.124847 1662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0: Neither blocks nor log segments found. Creating new log.
I20250109 21:09:02.125619 1662 log.cc:826] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0: Log is configured to *not* fsync() on all Append() calls
I20250109 21:09:02.127303 1662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0: No bootstrap required, opened a new log
I20250109 21:09:02.130090 1662 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a46131e086d47f894d1a951cf35f9b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35311 } }
I20250109 21:09:02.130276 1662 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:09:02.130363 1662 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a46131e086d47f894d1a951cf35f9b0, State: Initialized, Role: FOLLOWER
I20250109 21:09:02.130774 1662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a46131e086d47f894d1a951cf35f9b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35311 } }
I20250109 21:09:02.130931 1662 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:09:02.131002 1662 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:09:02.131094 1662 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0 [term 0 FOLLOWER]: Advancing to term 1
Jan 09 21:09:02 dist-test-slave-g0zb krb5kdc[1580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20250109 21:09:02.131851 1662 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a46131e086d47f894d1a951cf35f9b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35311 } }
I20250109 21:09:02.132145 1662 leader_election.cc:304] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a46131e086d47f894d1a951cf35f9b0; no voters:
I20250109 21:09:02.132455 1662 leader_election.cc:290] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:09:02.132542 1667 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:09:02.132757 1667 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0 [term 1 LEADER]: Becoming Leader. State: Replica: 0a46131e086d47f894d1a951cf35f9b0, State: Running, Role: LEADER
I20250109 21:09:02.133025 1667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a46131e086d47f894d1a951cf35f9b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35311 } }
I20250109 21:09:02.133522 1662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:09:02.134599 1668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a46131e086d47f894d1a951cf35f9b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a46131e086d47f894d1a951cf35f9b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35311 } } }
I20250109 21:09:02.134714 1668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0 [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:02.135123 1669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a46131e086d47f894d1a951cf35f9b0. Latest consensus state: current_term: 1 leader_uuid: "0a46131e086d47f894d1a951cf35f9b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a46131e086d47f894d1a951cf35f9b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35311 } } }
I20250109 21:09:02.135264 1669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0 [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:02.137432 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:02.125378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:35107 (local address 127.0.0.1:35311)
0109 21:09:02.125802 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:02.125812 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:02.125844 (+ 32us) server_negotiation.cc:407] Connection header received
0109 21:09:02.126528 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:02.126549 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:02.126874 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:02.127184 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:02.128011 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:02.128725 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:02.129353 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:02.129615 (+ 262us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:02.131888 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:02.131908 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:02.131919 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:02.131949 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:02.134524 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:02.135196 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:02.135204 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:02.135209 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:02.135300 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:02.135641 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:02.135644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:02.135645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:02.136076 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:02.136304 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:02.136541 (+ 237us) server_negotiation.cc:299] Negotiation successful
0109 21:09:02.136741 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":118,"threads_started":1}
W20250109 21:09:02.138082 1682 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:09:02.138160 1682 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 21:09:02.138401 1676 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:09:02.139278 1676 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:09:02.140482 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:09:02.144881 1676 catalog_manager.cc:1348] Generated new cluster ID: 9696239d83b84381814eb54190a77958
I20250109 21:09:02.144940 1676 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:09:02.151041 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:02.142326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:45967 (local address 127.0.0.1:35311)
0109 21:09:02.142465 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:02.142469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:02.143080 (+ 611us) server_negotiation.cc:407] Connection header received
0109 21:09:02.143239 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:02.143242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:02.143292 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:02.143370 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:02.144083 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:02.144507 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:02.145208 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:02.145404 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:02.146638 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:02.146654 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:02.146657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:02.146682 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:02.148385 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:02.148925 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:02.148930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:02.148931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:02.148996 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:02.149265 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:02.149269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:02.149271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:02.149438 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:02.149578 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:02.149782 (+ 204us) server_negotiation.cc:299] Negotiation successful
0109 21:09:02.149875 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250109 21:09:02.152765 1676 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:09:02.153827 1676 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:09:02.159969 1676 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 0a46131e086d47f894d1a951cf35f9b0: Generated new TSK 0
I20250109 21:09:02.160562 1676 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:09:02.169906 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 1593
W20250109 21:09:02.175360 1578 connection.cc:537] client connection to 127.0.0.1:35311 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35311): Connection reset by peer (error 104)
2025-01-09T21:09:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:09:02 dist-test-slave-g0zb krb5kdc[1693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:09:02 dist-test-slave-g0zb krb5kdc[1693](info): set up 2 sockets
Jan 09 21:09:02 dist-test-slave-g0zb krb5kdc[1693](info): commencing operation
krb5kdc: starting...
W20250109 21:09:03.155457 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:09:03 dist-test-slave-g0zb krb5kdc[1693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:09:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:09:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20250109 21:09:03.267483 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:09:03.371724 1706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:09:03.371994 1706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:09:03.372081 1706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:09:03.375195 1706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:09:03.375264 1706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:09:03.375296 1706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:09:03.375327 1706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:09:03.375362 1706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:09:03.379354 1706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39843
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:09:03.380419 1706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:09:03.381263 1706 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:09:03.388427 1714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:03.388617 1712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:03.388691 1711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:09:03.388931 1706 server_base.cc:1034] running on GCE node
I20250109 21:09:03.389433 1706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:09:03.390376 1706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:09:03.391544 1706 hybrid_clock.cc:648] HybridClock initialized: now 1736456943391534 us; error 24 us; skew 500 ppm
Jan 09 21:09:03 dist-test-slave-g0zb krb5kdc[1693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:09:03.394310 1706 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20250109 21:09:03.395294 1706 webserver.cc:458] Webserver started at http://127.0.0.1:34001/ using document root <none> and password file <none>
I20250109 21:09:03.395875 1706 fs_manager.cc:362] Metadata directory not provided
I20250109 21:09:03.395941 1706 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:09:03.396179 1706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:09:03.398160 1706 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "549777bb498f40468c178293a3ea7be5"
format_stamp: "Formatted at 2025-01-09 21:09:03 on dist-test-slave-g0zb"
server_key: "adba0939064d682e44c9f53f4eb3290a"
server_key_iv: "f9e325d401dc5e75265a37da110dd83d"
server_key_version: "encryptionkey@0"
I20250109 21:09:03.398663 1706 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "549777bb498f40468c178293a3ea7be5"
format_stamp: "Formatted at 2025-01-09 21:09:03 on dist-test-slave-g0zb"
server_key: "adba0939064d682e44c9f53f4eb3290a"
server_key_iv: "f9e325d401dc5e75265a37da110dd83d"
server_key_version: "encryptionkey@0"
I20250109 21:09:03.402061 1706 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250109 21:09:03.404253 1721 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:09:03.405256 1706 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:09:03.405354 1706 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "549777bb498f40468c178293a3ea7be5"
format_stamp: "Formatted at 2025-01-09 21:09:03 on dist-test-slave-g0zb"
server_key: "adba0939064d682e44c9f53f4eb3290a"
server_key_iv: "f9e325d401dc5e75265a37da110dd83d"
server_key_version: "encryptionkey@0"
I20250109 21:09:03.405458 1706 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:09:03.418921 1706 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:09:03.421897 1706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:09:03.422065 1706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:09:03.428659 1706 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39843
I20250109 21:09:03.428668 1773 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:39843 every 8 connection(s)
I20250109 21:09:03.429620 1706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:09:03.432624 1774 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:09:03.432667 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 1706
I20250109 21:09:03.432791 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:09:03.433053 12793 external_mini_cluster.cc:1453] Setting key 879023132c6742046ee3df1564990320
I20250109 21:09:03.438135 1774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5: Bootstrap starting.
I20250109 21:09:03.440106 1777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:03.434251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:40871 (local address 127.0.0.1:39843)
0109 21:09:03.434661 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:03.434670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:03.434702 (+ 32us) server_negotiation.cc:407] Connection header received
0109 21:09:03.435438 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:03.435457 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:03.435745 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:03.436065 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:03.436479 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:03.437172 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:03.437803 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:03.438071 (+ 268us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:03.438604 (+ 533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:03.438666 (+ 62us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:03.438676 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:09:03.439072 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:03.439105 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:09:03.439123 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:03.439235 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:03.439444 (+ 209us) server_negotiation.cc:299] Negotiation successful
0109 21:09:03.439569 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":140,"threads_started":1}
I20250109 21:09:03.440263 1774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5: Neither blocks nor log segments found. Creating new log.
I20250109 21:09:03.441088 1774 log.cc:826] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5: Log is configured to *not* fsync() on all Append() calls
I20250109 21:09:03.442605 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:09:03.443013 1774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5: No bootstrap required, opened a new log
I20250109 21:09:03.446841 1774 raft_consensus.cc:357] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549777bb498f40468c178293a3ea7be5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39843 } }
I20250109 21:09:03.447089 1774 raft_consensus.cc:383] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:09:03.447188 1774 raft_consensus.cc:738] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 549777bb498f40468c178293a3ea7be5, State: Initialized, Role: FOLLOWER
I20250109 21:09:03.447635 1774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549777bb498f40468c178293a3ea7be5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39843 } }
I20250109 21:09:03.447827 1777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:03.444697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:58445 (local address 127.0.0.1:39843)
0109 21:09:03.444815 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:03.444819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:03.444908 (+ 89us) server_negotiation.cc:407] Connection header received
0109 21:09:03.445004 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:03.445008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:03.445051 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:03.445142 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:03.445469 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:03.446164 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:03.446705 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:03.446926 (+ 221us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:03.447417 (+ 491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:03.447431 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:03.447433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:09:03.447445 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:03.447470 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:09:03.447481 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:03.447556 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:03.447639 (+ 83us) server_negotiation.cc:299] Negotiation successful
0109 21:09:03.447699 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250109 21:09:03.447844 1774 raft_consensus.cc:397] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:09:03.447904 1774 raft_consensus.cc:491] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:09:03.448026 1774 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:09:03.448899 1774 raft_consensus.cc:513] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549777bb498f40468c178293a3ea7be5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39843 } }
I20250109 21:09:03.449220 1774 leader_election.cc:304] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 549777bb498f40468c178293a3ea7be5; no voters:
I20250109 21:09:03.449455 1774 leader_election.cc:290] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:09:03.449561 1784 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:09:03.449761 1784 raft_consensus.cc:695] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5 [term 1 LEADER]: Becoming Leader. State: Replica: 549777bb498f40468c178293a3ea7be5, State: Running, Role: LEADER
I20250109 21:09:03.450141 1784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549777bb498f40468c178293a3ea7be5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39843 } }
I20250109 21:09:03.450601 1774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:09:03.451767 1785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "549777bb498f40468c178293a3ea7be5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549777bb498f40468c178293a3ea7be5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39843 } } }
I20250109 21:09:03.451889 1785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5 [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:03.452205 1786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 549777bb498f40468c178293a3ea7be5. Latest consensus state: current_term: 1 leader_uuid: "549777bb498f40468c178293a3ea7be5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "549777bb498f40468c178293a3ea7be5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39843 } } }
I20250109 21:09:03.452297 1793 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:09:03.452365 1786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5 [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:03.455425 1793 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:09:03.461041 1793 catalog_manager.cc:1348] Generated new cluster ID: f6446745d9a142f1967690a2f9e85f60
I20250109 21:09:03.461110 1793 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:09:03.478185 1793 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:09:03.479302 1793 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:09:03.488689 1793 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 549777bb498f40468c178293a3ea7be5: Generated new TSK 0
I20250109 21:09:03.489538 1793 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:09:03.498960 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 1706
W20250109 21:09:03.505033 1691 connection.cc:537] client connection to 127.0.0.1:39843 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39843): Connection reset by peer (error 104)
2025-01-09T21:09:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:09:03 dist-test-slave-g0zb krb5kdc[1806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:09:03 dist-test-slave-g0zb krb5kdc[1806](info): set up 2 sockets
Jan 09 21:09:03 dist-test-slave-g0zb krb5kdc[1806](info): commencing operation
krb5kdc: starting...
W20250109 21:09:04.491577 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:09:04 dist-test-slave-g0zb krb5kdc[1806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:09:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:09:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20250109 21:09:04.603515 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36585
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:09:04.702160 1819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:09:04.702391 1819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:09:04.702446 1819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:09:04.705264 1819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:09:04.705313 1819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:09:04.705332 1819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:09:04.705348 1819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:09:04.705380 1819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:09:04.709159 1819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36585
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:09:04.710116 1819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:09:04.710947 1819 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:09:04.717262 1827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:04.717442 1824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:04.717648 1825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:09:04.717957 1819 server_base.cc:1034] running on GCE node
I20250109 21:09:04.718382 1819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:09:04.719225 1819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:09:04.720366 1819 hybrid_clock.cc:648] HybridClock initialized: now 1736456944720358 us; error 21 us; skew 500 ppm
Jan 09 21:09:04 dist-test-slave-g0zb krb5kdc[1806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:09:04.728350 1819 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20250109 21:09:04.729169 1819 webserver.cc:458] Webserver started at http://127.0.0.1:38501/ using document root <none> and password file <none>
I20250109 21:09:04.729671 1819 fs_manager.cc:362] Metadata directory not provided
I20250109 21:09:04.729733 1819 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:09:04.729888 1819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:09:04.731505 1819 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "e08f9a56d649482dbc5aa6aa62c2b2ba"
format_stamp: "Formatted at 2025-01-09 21:09:04 on dist-test-slave-g0zb"
server_key: "d7a52fd07944cc6df3b6fc6f8ec2e0fa"
server_key_iv: "a2276613727a089d011db1bf18bd299d"
server_key_version: "encryptionkey@0"
I20250109 21:09:04.731931 1819 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "e08f9a56d649482dbc5aa6aa62c2b2ba"
format_stamp: "Formatted at 2025-01-09 21:09:04 on dist-test-slave-g0zb"
server_key: "d7a52fd07944cc6df3b6fc6f8ec2e0fa"
server_key_iv: "a2276613727a089d011db1bf18bd299d"
server_key_version: "encryptionkey@0"
I20250109 21:09:04.734874 1819 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250109 21:09:04.736765 1834 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:09:04.737682 1819 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250109 21:09:04.737792 1819 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "e08f9a56d649482dbc5aa6aa62c2b2ba"
format_stamp: "Formatted at 2025-01-09 21:09:04 on dist-test-slave-g0zb"
server_key: "d7a52fd07944cc6df3b6fc6f8ec2e0fa"
server_key_iv: "a2276613727a089d011db1bf18bd299d"
server_key_version: "encryptionkey@0"
I20250109 21:09:04.737869 1819 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:09:04.742376 1819 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:09:04.744845 1819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:09:04.744987 1819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:09:04.751137 1819 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36585
I20250109 21:09:04.751152 1887 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:36585 every 8 connection(s)
I20250109 21:09:04.751987 1819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:09:04.754496 1888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:09:04.758869 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 1819
I20250109 21:09:04.758996 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:09:04.759023 1888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba: Bootstrap starting.
I20250109 21:09:04.759235 12793 external_mini_cluster.cc:1453] Setting key fd8f05fa536ee647d99cd645a4e8cad0
I20250109 21:09:04.761199 1888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba: Neither blocks nor log segments found. Creating new log.
I20250109 21:09:04.761956 1888 log.cc:826] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba: Log is configured to *not* fsync() on all Append() calls
I20250109 21:09:04.763549 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:04.760301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:58059 (local address 127.0.0.1:36585)
0109 21:09:04.760696 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:04.760704 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:04.760736 (+ 32us) server_negotiation.cc:407] Connection header received
0109 21:09:04.761241 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:04.761263 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:04.761497 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:04.761792 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:04.762589 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:04.762596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:04.762606 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:09:04.762616 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:04.762653 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:09:04.762667 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:04.762783 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:04.762934 (+ 151us) server_negotiation.cc:299] Negotiation successful
0109 21:09:04.763059 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":102,"threads_started":1}
I20250109 21:09:04.764328 1888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba: No bootstrap required, opened a new log
I20250109 21:09:04.765887 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:09:04.767406 1888 raft_consensus.cc:357] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e08f9a56d649482dbc5aa6aa62c2b2ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36585 } }
I20250109 21:09:04.767642 1888 raft_consensus.cc:383] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:09:04.767760 1888 raft_consensus.cc:738] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08f9a56d649482dbc5aa6aa62c2b2ba, State: Initialized, Role: FOLLOWER
I20250109 21:09:04.768206 1888 consensus_queue.cc:260] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e08f9a56d649482dbc5aa6aa62c2b2ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36585 } }
I20250109 21:09:04.768330 1888 raft_consensus.cc:397] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:09:04.768393 1888 raft_consensus.cc:491] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:09:04.768503 1888 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:09:04.769235 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:04.767710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:55389 (local address 127.0.0.1:36585)
0109 21:09:04.767827 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:04.767833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:04.767959 (+ 126us) server_negotiation.cc:407] Connection header received
0109 21:09:04.768087 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:04.768090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:04.768123 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:04.768191 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:04.768877 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:04.768880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:04.768883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:09:04.768894 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:04.768922 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:09:04.768932 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:04.769007 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:04.769088 (+ 81us) server_negotiation.cc:299] Negotiation successful
0109 21:09:04.769124 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250109 21:09:04.769280 1888 raft_consensus.cc:513] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e08f9a56d649482dbc5aa6aa62c2b2ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36585 } }
I20250109 21:09:04.769584 1888 leader_election.cc:304] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e08f9a56d649482dbc5aa6aa62c2b2ba; no voters:
I20250109 21:09:04.769850 1888 leader_election.cc:290] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:09:04.769917 1899 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:09:04.770205 1899 raft_consensus.cc:695] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba [term 1 LEADER]: Becoming Leader. State: Replica: e08f9a56d649482dbc5aa6aa62c2b2ba, State: Running, Role: LEADER
I20250109 21:09:04.770481 1899 consensus_queue.cc:237] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e08f9a56d649482dbc5aa6aa62c2b2ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36585 } }
I20250109 21:09:04.770991 1888 sys_catalog.cc:564] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:09:04.772058 1901 sys_catalog.cc:455] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader e08f9a56d649482dbc5aa6aa62c2b2ba. Latest consensus state: current_term: 1 leader_uuid: "e08f9a56d649482dbc5aa6aa62c2b2ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e08f9a56d649482dbc5aa6aa62c2b2ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36585 } } }
I20250109 21:09:04.772073 1900 sys_catalog.cc:455] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e08f9a56d649482dbc5aa6aa62c2b2ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e08f9a56d649482dbc5aa6aa62c2b2ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36585 } } }
I20250109 21:09:04.772249 1901 sys_catalog.cc:458] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:04.772256 1900 sys_catalog.cc:458] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:04.773166 1908 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:09:04.775741 1908 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:09:04.779973 1908 catalog_manager.cc:1348] Generated new cluster ID: 8196ff9d4d1a4172af94890940c7af7b
I20250109 21:09:04.780040 1908 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:09:04.793682 1908 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:09:04.793931 1908 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:09:04.797972 1908 catalog_manager.cc:5899] T 00000000000000000000000000000000 P e08f9a56d649482dbc5aa6aa62c2b2ba: Generated new TSK 0
I20250109 21:09:04.798503 1908 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:09:04.801301 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 1819
2025-01-09T21:09:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:09:04 dist-test-slave-g0zb krb5kdc[1920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:09:04 dist-test-slave-g0zb krb5kdc[1920](info): set up 2 sockets
Jan 09 21:09:04 dist-test-slave-g0zb krb5kdc[1920](info): commencing operation
krb5kdc: starting...
W20250109 21:09:05.795446 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:09:05 dist-test-slave-g0zb krb5kdc[1920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:09:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:09:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:09:05.901938 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:43625
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:43625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:09:06.001662 1933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:09:06.001900 1933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:09:06.001992 1933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:09:06.004999 1933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:09:06.005056 1933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:09:06.005095 1933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:09:06.005132 1933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:09:06.005169 1933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:09:06.009073 1933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:43625
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:43625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:09:06.010015 1933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:09:06.010805 1933 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250109 21:09:06.017745 1933 server_base.cc:1034] running on GCE node
W20250109 21:09:06.017932 1938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:06.018052 1941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:06.018155 1939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:09:06.018570 1933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:09:06.019390 1933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:09:06.020531 1933 hybrid_clock.cc:648] HybridClock initialized: now 1736456946020516 us; error 28 us; skew 500 ppm
Jan 09 21:09:06 dist-test-slave-g0zb krb5kdc[1920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456946, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:09:06.023047 1933 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:09:06.023810 1933 webserver.cc:458] Webserver started at http://127.12.126.126:33777/ using document root <none> and password file <none>
I20250109 21:09:06.024303 1933 fs_manager.cc:362] Metadata directory not provided
I20250109 21:09:06.024359 1933 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:09:06.024514 1933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:09:06.026120 1933 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "c305eff492d94b43a9d7625d6f60cdec"
format_stamp: "Formatted at 2025-01-09 21:09:06 on dist-test-slave-g0zb"
server_key: "f6e58d09eec06245e79decf3ea0fdd3d"
server_key_iv: "b9c9f0e43726f7fa12ce77ba872e923d"
server_key_version: "encryptionkey@0"
I20250109 21:09:06.026554 1933 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "c305eff492d94b43a9d7625d6f60cdec"
format_stamp: "Formatted at 2025-01-09 21:09:06 on dist-test-slave-g0zb"
server_key: "f6e58d09eec06245e79decf3ea0fdd3d"
server_key_iv: "b9c9f0e43726f7fa12ce77ba872e923d"
server_key_version: "encryptionkey@0"
I20250109 21:09:06.029510 1933 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:09:06.031436 1948 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:09:06.032275 1933 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:09:06.032389 1933 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "c305eff492d94b43a9d7625d6f60cdec"
format_stamp: "Formatted at 2025-01-09 21:09:06 on dist-test-slave-g0zb"
server_key: "f6e58d09eec06245e79decf3ea0fdd3d"
server_key_iv: "b9c9f0e43726f7fa12ce77ba872e923d"
server_key_version: "encryptionkey@0"
I20250109 21:09:06.032464 1933 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:09:06.044080 1933 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:09:06.046298 1933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:09:06.046429 1933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:09:06.052112 1933 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:43625
I20250109 21:09:06.052129 2000 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:43625 every 8 connection(s)
I20250109 21:09:06.052860 1933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:09:06.055326 2001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:09:06.056515 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 1933
I20250109 21:09:06.056636 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:09:06.056890 12793 external_mini_cluster.cc:1453] Setting key dccfa723c4ea486fcdb7c6d9c025f717
I20250109 21:09:06.060900 2001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec: Bootstrap starting.
I20250109 21:09:06.063108 2001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec: Neither blocks nor log segments found. Creating new log.
I20250109 21:09:06.063755 2001 log.cc:826] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec: Log is configured to *not* fsync() on all Append() calls
Jan 09 21:09:06 dist-test-slave-g0zb krb5kdc[1920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:09:06.065666 2001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec: No bootstrap required, opened a new log
I20250109 21:09:06.068424 2001 raft_consensus.cc:357] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c305eff492d94b43a9d7625d6f60cdec" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 43625 } }
I20250109 21:09:06.068598 2001 raft_consensus.cc:383] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:09:06.068691 2001 raft_consensus.cc:738] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c305eff492d94b43a9d7625d6f60cdec, State: Initialized, Role: FOLLOWER
I20250109 21:09:06.069065 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:06.057968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47279 (local address 127.12.126.126:43625)
0109 21:09:06.058377 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:06.058389 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:06.058429 (+ 40us) server_negotiation.cc:407] Connection header received
0109 21:09:06.059168 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:06.059191 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:06.059531 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:06.059914 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:06.060666 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:06.061740 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:06.062355 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:06.062588 (+ 233us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:06.064728 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:06.064748 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:06.064886 (+ 138us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:06.064927 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:06.066625 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:06.067110 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:06.067115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:06.067118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:06.067183 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:06.067445 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:06.067448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:06.067450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:06.067756 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:06.067969 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:06.068266 (+ 297us) server_negotiation.cc:299] Negotiation successful
0109 21:09:06.068432 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":98,"threads_started":1}
I20250109 21:09:06.069108 2001 consensus_queue.cc:260] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c305eff492d94b43a9d7625d6f60cdec" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 43625 } }
I20250109 21:09:06.069280 2001 raft_consensus.cc:397] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:09:06.069340 2001 raft_consensus.cc:491] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:09:06.069442 2001 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:09:06.070325 2001 raft_consensus.cc:513] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c305eff492d94b43a9d7625d6f60cdec" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 43625 } }
I20250109 21:09:06.070637 2001 leader_election.cc:304] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c305eff492d94b43a9d7625d6f60cdec; no voters:
I20250109 21:09:06.070968 2001 leader_election.cc:290] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:09:06.071061 2006 raft_consensus.cc:2798] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:09:06.071285 2006 raft_consensus.cc:695] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec [term 1 LEADER]: Becoming Leader. State: Replica: c305eff492d94b43a9d7625d6f60cdec, State: Running, Role: LEADER
I20250109 21:09:06.071525 2006 consensus_queue.cc:237] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c305eff492d94b43a9d7625d6f60cdec" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 43625 } }
I20250109 21:09:06.071753 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:09:06.072090 2001 sys_catalog.cc:564] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:09:06.072639 2006 sys_catalog.cc:455] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec [sys.catalog]: SysCatalogTable state changed. Reason: New leader c305eff492d94b43a9d7625d6f60cdec. Latest consensus state: current_term: 1 leader_uuid: "c305eff492d94b43a9d7625d6f60cdec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c305eff492d94b43a9d7625d6f60cdec" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 43625 } } }
I20250109 21:09:06.072749 2006 sys_catalog.cc:458] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:06.073026 2007 sys_catalog.cc:455] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c305eff492d94b43a9d7625d6f60cdec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c305eff492d94b43a9d7625d6f60cdec" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 43625 } } }
I20250109 21:09:06.073103 2013 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:09:06.075838 2013 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:09:06.073101 2007 sys_catalog.cc:458] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:06.082204 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:06.073833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44093 (local address 127.12.126.126:43625)
0109 21:09:06.073930 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:06.073934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:06.074187 (+ 253us) server_negotiation.cc:407] Connection header received
0109 21:09:06.074299 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:06.074303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:06.074346 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:06.074407 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:06.076398 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:06.076834 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:06.077378 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:06.077582 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:06.078725 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:06.078743 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:06.078748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:06.078779 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:06.080566 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:06.081068 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:06.081072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:06.081074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:06.081127 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:06.081443 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:06.081446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:06.081448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:06.081577 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:06.081739 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:06.081996 (+ 257us) server_negotiation.cc:299] Negotiation successful
0109 21:09:06.082086 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250109 21:09:06.082935 2013 catalog_manager.cc:1348] Generated new cluster ID: dccfcf4708824198b0aaa2f7e009eb95
I20250109 21:09:06.082989 2013 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:09:06.099665 2013 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:09:06.100890 2013 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:09:06.114449 2013 catalog_manager.cc:5899] T 00000000000000000000000000000000 P c305eff492d94b43a9d7625d6f60cdec: Generated new TSK 0
I20250109 21:09:06.115250 2013 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:09:06.129209 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 1933
W20250109 21:09:06.134760 1918 connection.cc:537] client connection to 127.12.126.126:43625 recv error: Network error: failed to read from TLS socket (remote: 127.12.126.126:43625): Connection reset by peer (error 104)
2025-01-09T21:09:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:09:06 dist-test-slave-g0zb krb5kdc[2032](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:09:06 dist-test-slave-g0zb krb5kdc[2032](info): set up 2 sockets
Jan 09 21:09:06 dist-test-slave-g0zb krb5kdc[2032](info): commencing operation
krb5kdc: starting...
W20250109 21:09:07.119572 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:09:07 dist-test-slave-g0zb krb5kdc[2032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:09:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:09:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:09:07.237449 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:42699
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:40773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:42699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:09:07.343055 2045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:09:07.343312 2045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:09:07.343398 2045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:09:07.346499 2045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:09:07.346588 2045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:09:07.346625 2045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:09:07.346665 2045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:09:07.346705 2045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:09:07.350904 2045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:40773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:42699
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:42699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:09:07.351976 2045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:09:07.352838 2045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:09:07.360260 2050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:07.360423 2051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:07.360656 2053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:09:07.360713 2045 server_base.cc:1034] running on GCE node
I20250109 21:09:07.361218 2045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:09:07.362202 2045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:09:07.363365 2045 hybrid_clock.cc:648] HybridClock initialized: now 1736456947363342 us; error 39 us; skew 500 ppm
Jan 09 21:09:07 dist-test-slave-g0zb krb5kdc[2032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456947, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:09:07.366117 2045 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:09:07.367110 2045 webserver.cc:458] Webserver started at http://127.12.126.126:43003/ using document root <none> and password file <none>
I20250109 21:09:07.367672 2045 fs_manager.cc:362] Metadata directory not provided
I20250109 21:09:07.367729 2045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:09:07.367902 2045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:09:07.369767 2045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "b585a40d7ad74b33b37ec9fb3a056a6a"
format_stamp: "Formatted at 2025-01-09 21:09:07 on dist-test-slave-g0zb"
server_key: "9ecdab5d415b62f6dc5a205006672698"
server_key_iv: "124f8372f896f656c8611477c061858f"
server_key_version: "encryptionkey@0"
I20250109 21:09:07.370272 2045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "b585a40d7ad74b33b37ec9fb3a056a6a"
format_stamp: "Formatted at 2025-01-09 21:09:07 on dist-test-slave-g0zb"
server_key: "9ecdab5d415b62f6dc5a205006672698"
server_key_iv: "124f8372f896f656c8611477c061858f"
server_key_version: "encryptionkey@0"
I20250109 21:09:07.373834 2045 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250109 21:09:07.375977 2060 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:09:07.376924 2045 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 21:09:07.377043 2045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "b585a40d7ad74b33b37ec9fb3a056a6a"
format_stamp: "Formatted at 2025-01-09 21:09:07 on dist-test-slave-g0zb"
server_key: "9ecdab5d415b62f6dc5a205006672698"
server_key_iv: "124f8372f896f656c8611477c061858f"
server_key_version: "encryptionkey@0"
I20250109 21:09:07.377127 2045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:09:07.410672 2045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:09:07.413661 2045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:09:07.413856 2045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:09:07.420640 2045 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:42699
I20250109 21:09:07.420648 2112 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:42699 every 8 connection(s)
I20250109 21:09:07.421649 2045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:09:07.423314 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 2045
I20250109 21:09:07.423450 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:09:07.423710 12793 external_mini_cluster.cc:1453] Setting key b4e781776b7148dcf6700a7a2c4d0cb2
I20250109 21:09:07.424904 2113 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:09:07.431190 2113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a: Bootstrap starting.
I20250109 21:09:07.431631 2116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:07.424940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53043 (local address 127.12.126.126:42699)
0109 21:09:07.425343 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:07.425354 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:07.425394 (+ 40us) server_negotiation.cc:407] Connection header received
0109 21:09:07.426114 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:07.426137 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:07.426484 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:07.426888 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:07.427325 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:07.428514 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:07.429138 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:07.429396 (+ 258us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:07.429890 (+ 494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:07.429956 (+ 66us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:07.429971 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:09:07.430454 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:07.430492 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:09:07.430514 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:07.430652 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:07.430776 (+ 124us) server_negotiation.cc:299] Negotiation successful
0109 21:09:07.430922 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
I20250109 21:09:07.433562 2113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a: Neither blocks nor log segments found. Creating new log.
I20250109 21:09:07.434424 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:09:07.434401 2113 log.cc:826] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a: Log is configured to *not* fsync() on all Append() calls
I20250109 21:09:07.436499 2113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a: No bootstrap required, opened a new log
I20250109 21:09:07.439807 2116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:07.436512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57819 (local address 127.12.126.126:42699)
0109 21:09:07.436638 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:07.436642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:07.436837 (+ 195us) server_negotiation.cc:407] Connection header received
0109 21:09:07.436974 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:07.436977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:07.437016 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:07.437081 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:07.437456 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:07.437877 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:07.438437 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:07.438626 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:07.439319 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:07.439336 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:07.439339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:09:07.439351 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:07.439381 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:09:07.439393 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:07.439489 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:07.439588 (+ 99us) server_negotiation.cc:299] Negotiation successful
0109 21:09:07.439624 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250109 21:09:07.439644 2113 raft_consensus.cc:357] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b585a40d7ad74b33b37ec9fb3a056a6a" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42699 } }
I20250109 21:09:07.439893 2113 raft_consensus.cc:383] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:09:07.439945 2113 raft_consensus.cc:738] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b585a40d7ad74b33b37ec9fb3a056a6a, State: Initialized, Role: FOLLOWER
I20250109 21:09:07.440389 2113 consensus_queue.cc:260] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b585a40d7ad74b33b37ec9fb3a056a6a" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42699 } }
I20250109 21:09:07.440516 2113 raft_consensus.cc:397] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:09:07.440582 2113 raft_consensus.cc:491] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:09:07.440708 2113 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:09:07.441529 2113 raft_consensus.cc:513] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b585a40d7ad74b33b37ec9fb3a056a6a" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42699 } }
I20250109 21:09:07.441881 2113 leader_election.cc:304] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b585a40d7ad74b33b37ec9fb3a056a6a; no voters:
I20250109 21:09:07.442152 2113 leader_election.cc:290] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:09:07.442250 2124 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:09:07.442448 2124 raft_consensus.cc:695] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a [term 1 LEADER]: Becoming Leader. State: Replica: b585a40d7ad74b33b37ec9fb3a056a6a, State: Running, Role: LEADER
I20250109 21:09:07.442771 2124 consensus_queue.cc:237] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b585a40d7ad74b33b37ec9fb3a056a6a" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42699 } }
I20250109 21:09:07.443241 2113 sys_catalog.cc:564] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:09:07.444303 2126 sys_catalog.cc:455] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b585a40d7ad74b33b37ec9fb3a056a6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b585a40d7ad74b33b37ec9fb3a056a6a" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42699 } } }
I20250109 21:09:07.444437 2126 sys_catalog.cc:458] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:07.444818 2133 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:09:07.444783 2125 sys_catalog.cc:455] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b585a40d7ad74b33b37ec9fb3a056a6a. Latest consensus state: current_term: 1 leader_uuid: "b585a40d7ad74b33b37ec9fb3a056a6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b585a40d7ad74b33b37ec9fb3a056a6a" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42699 } } }
I20250109 21:09:07.444932 2125 sys_catalog.cc:458] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:07.448009 2133 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:09:07.453513 2133 catalog_manager.cc:1348] Generated new cluster ID: c00096db185b4f77838942a92f9f3c15
I20250109 21:09:07.453613 2133 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:09:07.463074 2133 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:09:07.464268 2133 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:09:07.473210 2133 catalog_manager.cc:5899] T 00000000000000000000000000000000 P b585a40d7ad74b33b37ec9fb3a056a6a: Generated new TSK 0
I20250109 21:09:07.473872 2133 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:09:07.485706 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 2045
W20250109 21:09:07.491366 2030 connection.cc:537] client connection to 127.12.126.126:42699 recv error: Network error: failed to read from TLS socket (remote: 127.12.126.126:42699): Connection reset by peer (error 104)
2025-01-09T21:09:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:09:07 dist-test-slave-g0zb krb5kdc[2145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:09:07 dist-test-slave-g0zb krb5kdc[2145](info): set up 2 sockets
Jan 09 21:09:07 dist-test-slave-g0zb krb5kdc[2145](info): commencing operation
krb5kdc: starting...
W20250109 21:09:08.471697 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:09:08 dist-test-slave-g0zb krb5kdc[2145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:09:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:09:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:09:08.579483 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:35051
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:35051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:09:08.682829 2158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:09:08.683112 2158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:09:08.683244 2158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:09:08.686321 2158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:09:08.686383 2158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:09:08.686417 2158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:09:08.686450 2158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:09:08.686511 2158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:09:08.690613 2158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:35051
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:35051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:09:08.691732 2158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:09:08.692576 2158 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:09:08.699568 2163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:08.699806 2165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:08.699899 2167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:09:08.700142 2158 server_base.cc:1034] running on GCE node
I20250109 21:09:08.700628 2158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:09:08.701505 2158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:09:08.702665 2158 hybrid_clock.cc:648] HybridClock initialized: now 1736456948702656 us; error 31 us; skew 500 ppm
Jan 09 21:09:08 dist-test-slave-g0zb krb5kdc[2145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456948, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:09:08.710175 2158 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:09:08.711218 2158 webserver.cc:458] Webserver started at http://127.12.126.126:42715/ using document root <none> and password file <none>
I20250109 21:09:08.711774 2158 fs_manager.cc:362] Metadata directory not provided
I20250109 21:09:08.711833 2158 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:09:08.712018 2158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:09:08.713742 2158 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "32c19d822ac440daaf18a967af358b4c"
format_stamp: "Formatted at 2025-01-09 21:09:08 on dist-test-slave-g0zb"
server_key: "b3c96360b6116b97aa2ea7c0bdc671e7"
server_key_iv: "4e3e6194e8261eaacedc96b23c13c4de"
server_key_version: "encryptionkey@0"
I20250109 21:09:08.714208 2158 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "32c19d822ac440daaf18a967af358b4c"
format_stamp: "Formatted at 2025-01-09 21:09:08 on dist-test-slave-g0zb"
server_key: "b3c96360b6116b97aa2ea7c0bdc671e7"
server_key_iv: "4e3e6194e8261eaacedc96b23c13c4de"
server_key_version: "encryptionkey@0"
I20250109 21:09:08.717587 2158 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:09:08.719693 2174 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:09:08.720685 2158 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:09:08.720791 2158 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "32c19d822ac440daaf18a967af358b4c"
format_stamp: "Formatted at 2025-01-09 21:09:08 on dist-test-slave-g0zb"
server_key: "b3c96360b6116b97aa2ea7c0bdc671e7"
server_key_iv: "4e3e6194e8261eaacedc96b23c13c4de"
server_key_version: "encryptionkey@0"
I20250109 21:09:08.720875 2158 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:09:08.725522 2158 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:09:08.728404 2158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:09:08.728554 2158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:09:08.734658 2226 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:35051 every 8 connection(s)
I20250109 21:09:08.734655 2158 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:35051
I20250109 21:09:08.735620 2158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:09:08.738353 2227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:09:08.743196 2227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c: Bootstrap starting.
I20250109 21:09:08.745338 2227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c: Neither blocks nor log segments found. Creating new log.
I20250109 21:09:08.745553 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 2158
I20250109 21:09:08.745683 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:09:08.745958 12793 external_mini_cluster.cc:1453] Setting key 99e3494a9c3b41bd80048dea97ec5bcd
I20250109 21:09:08.746237 2227 log.cc:826] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c: Log is configured to *not* fsync() on all Append() calls
I20250109 21:09:08.748140 2227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c: No bootstrap required, opened a new log
I20250109 21:09:08.750499 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:08.747052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34493 (local address 127.12.126.126:35051)
0109 21:09:08.747493 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:08.747507 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:08.747558 (+ 51us) server_negotiation.cc:407] Connection header received
0109 21:09:08.748139 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:08.748155 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:08.748423 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:08.748773 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:08.749539 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:08.749544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:08.749554 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:09:08.749564 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:08.749605 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:09:08.749619 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:08.749739 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:08.749819 (+ 80us) server_negotiation.cc:299] Negotiation successful
0109 21:09:08.749970 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":136,"threads_started":1}
I20250109 21:09:08.751502 2227 raft_consensus.cc:357] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c19d822ac440daaf18a967af358b4c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35051 } }
I20250109 21:09:08.751713 2227 raft_consensus.cc:383] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:09:08.751770 2227 raft_consensus.cc:738] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c19d822ac440daaf18a967af358b4c, State: Initialized, Role: FOLLOWER
I20250109 21:09:08.752213 2227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c19d822ac440daaf18a967af358b4c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35051 } }
I20250109 21:09:08.752346 2227 raft_consensus.cc:397] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:09:08.752434 2227 raft_consensus.cc:491] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:09:08.752559 2227 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:09:08.753100 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:09:08.753527 2227 raft_consensus.cc:513] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c19d822ac440daaf18a967af358b4c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35051 } }
I20250109 21:09:08.753854 2227 leader_election.cc:304] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32c19d822ac440daaf18a967af358b4c; no voters:
I20250109 21:09:08.754120 2227 leader_election.cc:290] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:09:08.754237 2232 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:09:08.754487 2232 raft_consensus.cc:695] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c [term 1 LEADER]: Becoming Leader. State: Replica: 32c19d822ac440daaf18a967af358b4c, State: Running, Role: LEADER
I20250109 21:09:08.754866 2232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c19d822ac440daaf18a967af358b4c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35051 } }
I20250109 21:09:08.755403 2227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:09:08.756505 2238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32c19d822ac440daaf18a967af358b4c. Latest consensus state: current_term: 1 leader_uuid: "32c19d822ac440daaf18a967af358b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c19d822ac440daaf18a967af358b4c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35051 } } }
I20250109 21:09:08.756628 2238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:08.756958 2246 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:09:08.757388 2237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32c19d822ac440daaf18a967af358b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c19d822ac440daaf18a967af358b4c" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 35051 } } }
I20250109 21:09:08.757521 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:08.755395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37151 (local address 127.12.126.126:35051)
0109 21:09:08.755519 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:08.755523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:08.755537 (+ 14us) server_negotiation.cc:407] Connection header received
0109 21:09:08.755573 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:08.755576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:08.755606 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:08.755688 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:08.757072 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:08.757075 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:08.757078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:09:08.757089 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:08.757115 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:09:08.757126 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:08.757210 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:08.757361 (+ 151us) server_negotiation.cc:299] Negotiation successful
0109 21:09:08.757403 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250109 21:09:08.759056 2237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:08.759908 2246 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:09:08.765126 2246 catalog_manager.cc:1348] Generated new cluster ID: 42e8250749254af3aa76530bc9e1333e
I20250109 21:09:08.765197 2246 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:09:08.779570 2246 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:09:08.779944 2246 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:09:08.794945 2246 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 32c19d822ac440daaf18a967af358b4c: Generated new TSK 0
I20250109 21:09:08.795668 2246 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:09:08.806443 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 2158
2025-01-09T21:09:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:09:08 dist-test-slave-g0zb krb5kdc[2258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:09:08 dist-test-slave-g0zb krb5kdc[2258](info): set up 2 sockets
Jan 09 21:09:08 dist-test-slave-g0zb krb5kdc[2258](info): commencing operation
krb5kdc: starting...
W20250109 21:09:09.819718 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:09:09 dist-test-slave-g0zb krb5kdc[2258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:09:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:09:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:09:09.931883 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:44793
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:37759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:44793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:09:10.038100 2271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:09:10.038378 2271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:09:10.038465 2271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:09:10.041615 2271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:09:10.041824 2271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:09:10.041942 2271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:09:10.041983 2271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:09:10.042027 2271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:09:10.046375 2271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:37759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:44793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:44793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:09:10.047554 2271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:09:10.048439 2271 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:09:10.055696 2279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:10.055830 2276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:10.056063 2277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:09:10.056316 2271 server_base.cc:1034] running on GCE node
I20250109 21:09:10.056782 2271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:09:10.057687 2271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:09:10.058861 2271 hybrid_clock.cc:648] HybridClock initialized: now 1736456950058846 us; error 31 us; skew 500 ppm
Jan 09 21:09:10 dist-test-slave-g0zb krb5kdc[2258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456950, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:09:10.068573 2271 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:09:10.069617 2271 webserver.cc:458] Webserver started at http://127.12.126.126:45153/ using document root <none> and password file <none>
I20250109 21:09:10.070209 2271 fs_manager.cc:362] Metadata directory not provided
I20250109 21:09:10.070266 2271 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:09:10.070430 2271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:09:10.072324 2271 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "a99a820e094f43aca47f1766db190e9b"
format_stamp: "Formatted at 2025-01-09 21:09:10 on dist-test-slave-g0zb"
server_key: "31b3545c9078621bce3fbd31e6974465"
server_key_iv: "57df94b87c4f4121a62ed91b97c8b83a"
server_key_version: "encryptionkey@0"
I20250109 21:09:10.072803 2271 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "a99a820e094f43aca47f1766db190e9b"
format_stamp: "Formatted at 2025-01-09 21:09:10 on dist-test-slave-g0zb"
server_key: "31b3545c9078621bce3fbd31e6974465"
server_key_iv: "57df94b87c4f4121a62ed91b97c8b83a"
server_key_version: "encryptionkey@0"
I20250109 21:09:10.076294 2271 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20250109 21:09:10.078430 2286 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:09:10.079389 2271 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 21:09:10.079493 2271 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "a99a820e094f43aca47f1766db190e9b"
format_stamp: "Formatted at 2025-01-09 21:09:10 on dist-test-slave-g0zb"
server_key: "31b3545c9078621bce3fbd31e6974465"
server_key_iv: "57df94b87c4f4121a62ed91b97c8b83a"
server_key_version: "encryptionkey@0"
I20250109 21:09:10.079578 2271 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:09:10.094592 2271 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:09:10.097290 2271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:09:10.097471 2271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:09:10.104710 2271 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:44793
I20250109 21:09:10.104719 2339 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:44793 every 8 connection(s)
I20250109 21:09:10.105715 2271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:09:10.107704 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 2271
I20250109 21:09:10.107836 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:09:10.108099 12793 external_mini_cluster.cc:1453] Setting key 1b997e76ba524831e415971bccbd6e4f
I20250109 21:09:10.109050 2340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:09:10.116194 2340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b: Bootstrap starting.
Jan 09 21:09:10 dist-test-slave-g0zb krb5kdc[2258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:09:10.118531 2340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b: Neither blocks nor log segments found. Creating new log.
I20250109 21:09:10.119277 2340 log.cc:826] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b: Log is configured to *not* fsync() on all Append() calls
I20250109 21:09:10.121491 2340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b: No bootstrap required, opened a new log
I20250109 21:09:10.121804 2343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:10.109342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44371 (local address 127.12.126.126:44793)
0109 21:09:10.109729 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:10.109738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:10.109774 (+ 36us) server_negotiation.cc:407] Connection header received
0109 21:09:10.110380 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:10.110398 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:10.110697 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:10.111086 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:10.112234 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:10.113385 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:10.114361 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:10.114682 (+ 321us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:10.117132 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:10.117172 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:10.117185 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:10.117224 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:10.119094 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:10.119517 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:10.119523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:10.119529 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:10.119633 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:10.119883 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:10.119885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:10.119886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:10.120322 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:10.120530 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:10.120769 (+ 239us) server_negotiation.cc:299] Negotiation successful
0109 21:09:10.121044 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":118,"threads_started":1}
I20250109 21:09:10.124408 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:09:10.124536 2340 raft_consensus.cc:357] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a99a820e094f43aca47f1766db190e9b" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 44793 } }
I20250109 21:09:10.124728 2340 raft_consensus.cc:383] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:09:10.124784 2340 raft_consensus.cc:738] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a99a820e094f43aca47f1766db190e9b, State: Initialized, Role: FOLLOWER
I20250109 21:09:10.125206 2340 consensus_queue.cc:260] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a99a820e094f43aca47f1766db190e9b" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 44793 } }
I20250109 21:09:10.125319 2340 raft_consensus.cc:397] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:09:10.125388 2340 raft_consensus.cc:491] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:09:10.125479 2340 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:09:10.126483 2340 raft_consensus.cc:513] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a99a820e094f43aca47f1766db190e9b" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 44793 } }
I20250109 21:09:10.126835 2340 leader_election.cc:304] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a99a820e094f43aca47f1766db190e9b; no voters:
I20250109 21:09:10.127110 2340 leader_election.cc:290] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:09:10.127252 2350 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:09:10.127451 2350 raft_consensus.cc:695] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b [term 1 LEADER]: Becoming Leader. State: Replica: a99a820e094f43aca47f1766db190e9b, State: Running, Role: LEADER
I20250109 21:09:10.127820 2350 consensus_queue.cc:237] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a99a820e094f43aca47f1766db190e9b" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 44793 } }
I20250109 21:09:10.128249 2340 sys_catalog.cc:564] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:09:10.129148 2352 sys_catalog.cc:455] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a99a820e094f43aca47f1766db190e9b. Latest consensus state: current_term: 1 leader_uuid: "a99a820e094f43aca47f1766db190e9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a99a820e094f43aca47f1766db190e9b" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 44793 } } }
I20250109 21:09:10.129262 2352 sys_catalog.cc:458] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:10.129585 2359 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:09:10.129738 2351 sys_catalog.cc:455] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a99a820e094f43aca47f1766db190e9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a99a820e094f43aca47f1766db190e9b" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 44793 } } }
I20250109 21:09:10.129899 2351 sys_catalog.cc:458] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:10.132491 2359 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:09:10.134346 2343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:10.126271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40699 (local address 127.12.126.126:44793)
0109 21:09:10.126365 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:10.126368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:10.126562 (+ 194us) server_negotiation.cc:407] Connection header received
0109 21:09:10.126671 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:10.126674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:10.126729 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:10.126826 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:10.127669 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:10.128308 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:10.129320 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:10.129526 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:10.131440 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:10.131456 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:10.131459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:10.131486 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:10.132957 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:10.133370 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:10.133373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:10.133375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:10.133418 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:10.133632 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:10.133635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:10.133637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:10.133772 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:10.133905 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:10.134091 (+ 186us) server_negotiation.cc:299] Negotiation successful
0109 21:09:10.134196 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20250109 21:09:10.137861 2359 catalog_manager.cc:1348] Generated new cluster ID: 5e78e278115b43a1a120206abf9da5d1
I20250109 21:09:10.137923 2359 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:09:10.161551 2359 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:09:10.162535 2359 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:09:10.171483 2359 catalog_manager.cc:5899] T 00000000000000000000000000000000 P a99a820e094f43aca47f1766db190e9b: Generated new TSK 0
I20250109 21:09:10.172215 2359 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:09:10.176826 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 2271
2025-01-09T21:09:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:09:10 dist-test-slave-g0zb krb5kdc[2371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:09:10 dist-test-slave-g0zb krb5kdc[2371](info): set up 2 sockets
Jan 09 21:09:10 dist-test-slave-g0zb krb5kdc[2371](info): commencing operation
krb5kdc: starting...
W20250109 21:09:11.167555 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:09:11 dist-test-slave-g0zb krb5kdc[2371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:09:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:09:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:09:11.275005 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:34591
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:34591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:09:11.375324 2384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:09:11.375566 2384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:09:11.375645 2384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:09:11.378731 2384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:09:11.378793 2384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:09:11.378865 2384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:09:11.378914 2384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:09:11.378950 2384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:09:11.383060 2384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:34591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:34591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:09:11.384089 2384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:09:11.384917 2384 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:09:11.391757 2389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:11.391990 2390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:11.392222 2392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:09:11.392468 2384 server_base.cc:1034] running on GCE node
I20250109 21:09:11.392912 2384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:09:11.393774 2384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:09:11.394904 2384 hybrid_clock.cc:648] HybridClock initialized: now 1736456951394888 us; error 36 us; skew 500 ppm
Jan 09 21:09:11 dist-test-slave-g0zb krb5kdc[2371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456951, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:09:11.397512 2384 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:09:11.398361 2384 webserver.cc:458] Webserver started at http://127.12.126.126:41639/ using document root <none> and password file <none>
I20250109 21:09:11.398905 2384 fs_manager.cc:362] Metadata directory not provided
I20250109 21:09:11.398977 2384 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:09:11.399130 2384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:09:11.400858 2384 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "b9a1c5be384b4b0ebddda192d51af1b4"
format_stamp: "Formatted at 2025-01-09 21:09:11 on dist-test-slave-g0zb"
server_key: "ecccc7acbc2b3e4cd4f18700198eccd8"
server_key_iv: "fd81eb28457a27b8a3d4b064a726c319"
server_key_version: "encryptionkey@0"
I20250109 21:09:11.401289 2384 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "b9a1c5be384b4b0ebddda192d51af1b4"
format_stamp: "Formatted at 2025-01-09 21:09:11 on dist-test-slave-g0zb"
server_key: "ecccc7acbc2b3e4cd4f18700198eccd8"
server_key_iv: "fd81eb28457a27b8a3d4b064a726c319"
server_key_version: "encryptionkey@0"
I20250109 21:09:11.404577 2384 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:09:11.406526 2399 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:09:11.407490 2384 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:09:11.407588 2384 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "b9a1c5be384b4b0ebddda192d51af1b4"
format_stamp: "Formatted at 2025-01-09 21:09:11 on dist-test-slave-g0zb"
server_key: "ecccc7acbc2b3e4cd4f18700198eccd8"
server_key_iv: "fd81eb28457a27b8a3d4b064a726c319"
server_key_version: "encryptionkey@0"
I20250109 21:09:11.407673 2384 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:09:11.423503 2384 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:09:11.426434 2384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:09:11.426599 2384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:09:11.432930 2384 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:34591
I20250109 21:09:11.432950 2451 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:34591 every 8 connection(s)
I20250109 21:09:11.433887 2384 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:09:11.436797 2452 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:09:11.439589 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 2384
I20250109 21:09:11.439728 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:09:11.439975 12793 external_mini_cluster.cc:1453] Setting key c6e6ed8696011466fedbad2a33a4e6f2
I20250109 21:09:11.441939 2452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4: Bootstrap starting.
I20250109 21:09:11.443962 2452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4: Neither blocks nor log segments found. Creating new log.
I20250109 21:09:11.444605 2452 log.cc:826] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4: Log is configured to *not* fsync() on all Append() calls
I20250109 21:09:11.446427 2452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4: No bootstrap required, opened a new log
I20250109 21:09:11.447094 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:11.441216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49013 (local address 127.12.126.126:34591)
0109 21:09:11.441628 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:11.441637 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:11.441669 (+ 32us) server_negotiation.cc:407] Connection header received
0109 21:09:11.442250 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:11.442269 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:11.442549 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:11.442902 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:11.443391 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:11.444168 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:11.444821 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:11.445090 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:11.445614 (+ 524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:11.445630 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:11.445640 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:09:11.445990 (+ 350us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:11.446022 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:09:11.446039 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:11.446122 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:11.446260 (+ 138us) server_negotiation.cc:299] Negotiation successful
0109 21:09:11.446405 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":135,"threads_started":1}
I20250109 21:09:11.449684 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:09:11.449681 2452 raft_consensus.cc:357] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a1c5be384b4b0ebddda192d51af1b4" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 34591 } }
I20250109 21:09:11.449889 2452 raft_consensus.cc:383] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:09:11.449951 2452 raft_consensus.cc:738] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9a1c5be384b4b0ebddda192d51af1b4, State: Initialized, Role: FOLLOWER
I20250109 21:09:11.450417 2452 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a1c5be384b4b0ebddda192d51af1b4" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 34591 } }
I20250109 21:09:11.450575 2452 raft_consensus.cc:397] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:09:11.450641 2452 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:09:11.450740 2452 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:09:11.451838 2452 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a1c5be384b4b0ebddda192d51af1b4" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 34591 } }
I20250109 21:09:11.452266 2452 leader_election.cc:304] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9a1c5be384b4b0ebddda192d51af1b4; no voters:
I20250109 21:09:11.452574 2452 leader_election.cc:290] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:09:11.452648 2462 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:09:11.452848 2462 raft_consensus.cc:695] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4 [term 1 LEADER]: Becoming Leader. State: Replica: b9a1c5be384b4b0ebddda192d51af1b4, State: Running, Role: LEADER
I20250109 21:09:11.453168 2462 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a1c5be384b4b0ebddda192d51af1b4" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 34591 } }
I20250109 21:09:11.453722 2452 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:09:11.454145 2462 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9a1c5be384b4b0ebddda192d51af1b4. Latest consensus state: current_term: 1 leader_uuid: "b9a1c5be384b4b0ebddda192d51af1b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a1c5be384b4b0ebddda192d51af1b4" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 34591 } } }
I20250109 21:09:11.454280 2462 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4 [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:11.454679 2463 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9a1c5be384b4b0ebddda192d51af1b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a1c5be384b4b0ebddda192d51af1b4" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 34591 } } }
I20250109 21:09:11.454838 2463 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4 [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:11.455541 2466 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:09:11.456063 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:11.451907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51865 (local address 127.12.126.126:34591)
0109 21:09:11.452020 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:11.452024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:11.452127 (+ 103us) server_negotiation.cc:407] Connection header received
0109 21:09:11.452255 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:11.452257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:11.452295 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:11.452383 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:11.452861 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:11.453519 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:11.454330 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:11.454559 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:11.455541 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:11.455561 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:11.455564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:09:11.455576 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:11.455604 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:09:11.455617 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:11.455665 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:11.455872 (+ 207us) server_negotiation.cc:299] Negotiation successful
0109 21:09:11.455931 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250109 21:09:11.458540 2466 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:09:11.463759 2466 catalog_manager.cc:1348] Generated new cluster ID: ea82bf331ae748d482310dda476e019e
I20250109 21:09:11.463961 2466 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:09:11.485770 2466 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:09:11.486654 2466 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:09:11.495998 2466 catalog_manager.cc:5899] T 00000000000000000000000000000000 P b9a1c5be384b4b0ebddda192d51af1b4: Generated new TSK 0
I20250109 21:09:11.496640 2466 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:09:11.505913 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 2384
2025-01-09T21:09:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:09:11 dist-test-slave-g0zb krb5kdc[2483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:09:11 dist-test-slave-g0zb krb5kdc[2483](info): set up 2 sockets
Jan 09 21:09:11 dist-test-slave-g0zb krb5kdc[2483](info): commencing operation
krb5kdc: starting...
W20250109 21:09:12.499686 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:09:12 dist-test-slave-g0zb krb5kdc[2483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:09:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:09:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:09:12.611101 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:45769
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:36341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:45769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:09:12.716673 2496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:09:12.716964 2496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:09:12.717072 2496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:09:12.720206 2496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:09:12.720299 2496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:09:12.720335 2496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:09:12.720381 2496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:09:12.720417 2496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:09:12.724763 2496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:36341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:45769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:45769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:09:12.725929 2496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:09:12.726856 2496 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:09:12.733757 2504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:12.733881 2501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:12.734053 2502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:09:12.734311 2496 server_base.cc:1034] running on GCE node
I20250109 21:09:12.734774 2496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:09:12.735731 2496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:09:12.736886 2496 hybrid_clock.cc:648] HybridClock initialized: now 1736456952736861 us; error 40 us; skew 500 ppm
Jan 09 21:09:12 dist-test-slave-g0zb krb5kdc[2483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456952, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:09:12.739645 2496 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:09:12.740574 2496 webserver.cc:458] Webserver started at http://127.12.126.126:39619/ using document root <none> and password file <none>
I20250109 21:09:12.741132 2496 fs_manager.cc:362] Metadata directory not provided
I20250109 21:09:12.741191 2496 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:09:12.741379 2496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:09:12.743296 2496 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "74b89c1414814302be9d8d60fb9bf5c1"
format_stamp: "Formatted at 2025-01-09 21:09:12 on dist-test-slave-g0zb"
server_key: "1a2c7426ea0f4d8c9bebab2654640e7a"
server_key_iv: "e54854ae839900a8e8bf935aca764ba2"
server_key_version: "encryptionkey@0"
I20250109 21:09:12.743785 2496 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "74b89c1414814302be9d8d60fb9bf5c1"
format_stamp: "Formatted at 2025-01-09 21:09:12 on dist-test-slave-g0zb"
server_key: "1a2c7426ea0f4d8c9bebab2654640e7a"
server_key_iv: "e54854ae839900a8e8bf935aca764ba2"
server_key_version: "encryptionkey@0"
I20250109 21:09:12.747071 2496 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250109 21:09:12.749269 2511 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:09:12.750272 2496 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:09:12.750384 2496 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "74b89c1414814302be9d8d60fb9bf5c1"
format_stamp: "Formatted at 2025-01-09 21:09:12 on dist-test-slave-g0zb"
server_key: "1a2c7426ea0f4d8c9bebab2654640e7a"
server_key_iv: "e54854ae839900a8e8bf935aca764ba2"
server_key_version: "encryptionkey@0"
I20250109 21:09:12.750476 2496 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:09:12.755291 2496 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:09:12.758219 2496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:09:12.758389 2496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:09:12.764804 2496 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:45769
I20250109 21:09:12.764812 2563 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:45769 every 8 connection(s)
I20250109 21:09:12.765794 2496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:09:12.766510 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 2496
I20250109 21:09:12.766623 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:09:12.766884 12793 external_mini_cluster.cc:1453] Setting key 30065e0cc02567a6b1c1810c7e4e2450
I20250109 21:09:12.768572 2564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:09:12.772578 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:12.768595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45365 (local address 127.12.126.126:45769)
0109 21:09:12.769016 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:12.769027 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:12.769068 (+ 41us) server_negotiation.cc:407] Connection header received
0109 21:09:12.769823 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:12.769847 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:12.770167 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:12.770595 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:12.771374 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:12.771381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:12.771395 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:09:12.771408 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:12.771476 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:09:12.771494 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:12.771678 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:12.771758 (+ 80us) server_negotiation.cc:299] Negotiation successful
0109 21:09:12.771893 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":136,"threads_started":1}
I20250109 21:09:12.774993 2564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1: Bootstrap starting.
I20250109 21:09:12.775223 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:09:12.777640 2564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1: Neither blocks nor log segments found. Creating new log.
I20250109 21:09:12.778518 2564 log.cc:826] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1: Log is configured to *not* fsync() on all Append() calls
I20250109 21:09:12.779289 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:12.777522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49189 (local address 127.12.126.126:45769)
0109 21:09:12.777653 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:12.777667 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:12.777918 (+ 251us) server_negotiation.cc:407] Connection header received
0109 21:09:12.778044 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:12.778046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:12.778081 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:12.778151 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:12.778906 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:12.778909 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:12.778912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0109 21:09:12.778923 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:12.778949 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0109 21:09:12.778961 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:12.779032 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:12.779118 (+ 86us) server_negotiation.cc:299] Negotiation successful
0109 21:09:12.779156 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250109 21:09:12.780380 2564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1: No bootstrap required, opened a new log
I20250109 21:09:12.783442 2564 raft_consensus.cc:357] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b89c1414814302be9d8d60fb9bf5c1" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 45769 } }
I20250109 21:09:12.783624 2564 raft_consensus.cc:383] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 21:09:12.783690 2564 raft_consensus.cc:738] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b89c1414814302be9d8d60fb9bf5c1, State: Initialized, Role: FOLLOWER
I20250109 21:09:12.784117 2564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b89c1414814302be9d8d60fb9bf5c1" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 45769 } }
I20250109 21:09:12.784250 2564 raft_consensus.cc:397] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250109 21:09:12.784309 2564 raft_consensus.cc:491] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250109 21:09:12.784409 2564 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:09:12.785410 2564 raft_consensus.cc:513] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b89c1414814302be9d8d60fb9bf5c1" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 45769 } }
I20250109 21:09:12.785746 2564 leader_election.cc:304] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74b89c1414814302be9d8d60fb9bf5c1; no voters:
I20250109 21:09:12.786013 2564 leader_election.cc:290] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250109 21:09:12.786105 2574 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:09:12.786314 2574 raft_consensus.cc:695] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 74b89c1414814302be9d8d60fb9bf5c1, State: Running, Role: LEADER
I20250109 21:09:12.786612 2574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b89c1414814302be9d8d60fb9bf5c1" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 45769 } }
I20250109 21:09:12.787120 2564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:09:12.788414 2576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74b89c1414814302be9d8d60fb9bf5c1. Latest consensus state: current_term: 1 leader_uuid: "74b89c1414814302be9d8d60fb9bf5c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b89c1414814302be9d8d60fb9bf5c1" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 45769 } } }
I20250109 21:09:12.788447 2575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74b89c1414814302be9d8d60fb9bf5c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b89c1414814302be9d8d60fb9bf5c1" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 45769 } } }
I20250109 21:09:12.788547 2576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1 [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:12.788555 2575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1 [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:12.789182 2583 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:09:12.791872 2583 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:09:12.796810 2583 catalog_manager.cc:1348] Generated new cluster ID: 03a13a3d88a7437da5252ea634bc2fb2
I20250109 21:09:12.796890 2583 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:09:12.817409 2583 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:09:12.817732 2583 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:09:12.833142 2583 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 74b89c1414814302be9d8d60fb9bf5c1: Generated new TSK 0
I20250109 21:09:12.834022 2583 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 21:09:12.849232 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 2496
2025-01-09T21:09:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1343 ms)
[----------] 18 tests from AuthTokenIssuingTest (24032 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:09:12 dist-test-slave-g0zb krb5kdc[2595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:09:12 dist-test-slave-g0zb krb5kdc[2595](info): set up 2 sockets
Jan 09 21:09:12 dist-test-slave-g0zb krb5kdc[2595](info): commencing operation
krb5kdc: starting...
W20250109 21:09:13.823781 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:09:13 dist-test-slave-g0zb krb5kdc[2595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:09:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:09:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:09:13.934525 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:41873
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:41873,127.12.126.125:45121,127.12.126.124:42171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:09:14.042531 2608 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250109 21:09:14.042778 2608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:09:14.042845 2608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:09:14.042932 2608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:09:14.045840 2608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:09:14.045905 2608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:09:14.045938 2608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:09:14.045976 2608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:09:14.045993 2608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:09:14.050009 2608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:41873,127.12.126.125:45121,127.12.126.124:42171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:41873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:09:14.051091 2608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:09:14.051927 2608 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:09:14.058264 2613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:14.058471 2616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:14.058668 2614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:09:14.058980 2608 server_base.cc:1034] running on GCE node
I20250109 21:09:14.059464 2608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:09:14.060364 2608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:09:14.061527 2608 hybrid_clock.cc:648] HybridClock initialized: now 1736456954061516 us; error 33 us; skew 500 ppm
Jan 09 21:09:14 dist-test-slave-g0zb krb5kdc[2595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456954, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:09:14.064126 2608 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:09:14.065099 2608 webserver.cc:458] Webserver started at http://127.12.126.126:38337/ using document root <none> and password file <none>
I20250109 21:09:14.065639 2608 fs_manager.cc:362] Metadata directory not provided
I20250109 21:09:14.065698 2608 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:09:14.065876 2608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:09:14.067723 2608 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "885708b5be4045da8a4eb11deefb566e"
format_stamp: "Formatted at 2025-01-09 21:09:14 on dist-test-slave-g0zb"
server_key: "97a394d493d5494978f66df7cdd3ed55"
server_key_iv: "a13d203a0f6ad1e04d3b96c009b7d9af"
server_key_version: "encryptionkey@0"
I20250109 21:09:14.068240 2608 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "885708b5be4045da8a4eb11deefb566e"
format_stamp: "Formatted at 2025-01-09 21:09:14 on dist-test-slave-g0zb"
server_key: "97a394d493d5494978f66df7cdd3ed55"
server_key_iv: "a13d203a0f6ad1e04d3b96c009b7d9af"
server_key_version: "encryptionkey@0"
I20250109 21:09:14.071573 2608 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250109 21:09:14.073601 2623 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:09:14.074550 2608 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 21:09:14.074656 2608 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "885708b5be4045da8a4eb11deefb566e"
format_stamp: "Formatted at 2025-01-09 21:09:14 on dist-test-slave-g0zb"
server_key: "97a394d493d5494978f66df7cdd3ed55"
server_key_iv: "a13d203a0f6ad1e04d3b96c009b7d9af"
server_key_version: "encryptionkey@0"
I20250109 21:09:14.074739 2608 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:09:14.093600 2608 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:09:14.096567 2608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:09:14.096733 2608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:09:14.103475 2608 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:41873
I20250109 21:09:14.103487 2675 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:41873 every 8 connection(s)
I20250109 21:09:14.104470 2608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:09:14.107385 2676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:09:14.110664 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 2608
I20250109 21:09:14.110832 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:09:14.110704 2676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } has no permanent_uuid. Determining permanent_uuid...
I20250109 21:09:14.111099 12793 external_mini_cluster.cc:1453] Setting key bd89befeb9ff636352dc47dde7f9c77f
WARNING: no policy specified for kudu/127.12.126.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.125@KRBTEST.COM" created.
Jan 09 21:09:14 dist-test-slave-g0zb krb5kdc[2595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456954, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:09:14.127306 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.112861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58360 (local address 127.12.126.126:41873)
0109 21:09:14.113383 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.113394 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.113575 (+ 181us) server_negotiation.cc:407] Connection header received
0109 21:09:14.114885 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.114890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.115054 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.115206 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.116548 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.117240 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.118097 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.118256 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.122220 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:14.122249 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:14.122266 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:14.122300 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:14.124164 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.124743 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.124747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.124749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.124810 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.125115 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.125117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.125118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.125432 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:14.125549 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:14.126086 (+ 537us) server_negotiation.cc:299] Negotiation successful
0109 21:09:14.126310 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"spinlock_wait_cycles":58112,"thread_start_us":142,"threads_started":1}
I20250109 21:09:14.127307 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.112866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:41873 (local address 127.0.0.1:58360)
0109 21:09:14.113420 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.113467 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0109 21:09:14.114507 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:09:14.115262 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:09:14.115270 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:09:14.115894 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:09:14.116374 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:14.116393 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.117360 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.117364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:09:14.117982 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:14.117995 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.118255 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.118917 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:09:14.118958 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:09:14.122068 (+ 3110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:09:14.124299 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:14.124310 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:14.124323 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:14.124627 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:14.124892 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:14.124898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:14.124903 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:14.125017 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:14.125554 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:09:14.125563 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:09:14.125751 (+ 188us) client_negotiation.cc:769] Sending connection context
0109 21:09:14.126006 (+ 255us) client_negotiation.cc:240] Negotiation successful
0109 21:09:14.126422 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":112,"thread_start_us":142,"threads_started":1}
I20250109 21:09:14.130641 2676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } has no permanent_uuid. Determining permanent_uuid...
I20250109 21:09:14.131395 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.131134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:45121 (local address 127.0.0.1:37688)
0109 21:09:14.131264 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.131330 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.125:45121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20250109 21:09:14.131557 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.126.125:45121: connect: Connection refused (error 111)
I20250109 21:09:14.132378 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.132192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:45121 (local address 127.0.0.1:37700)
0109 21:09:14.132288 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.132329 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.125:45121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20250109 21:09:14.132488 2676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.125:45121: Network error: Client connection negotiation failed: client connection to 127.12.126.125:45121: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.125.keytab.
Entry for principal kudu/127.12.126.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.125.keytab.
I20250109 21:09:14.137362 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.125:45121
--webserver_interface=127.12.126.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.125.keytab
--principal=kudu/127.12.126.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:41873,127.12.126.125:45121,127.12.126.124:42171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:09:14.181779 2676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } attempt: 1
I20250109 21:09:14.182623 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.182333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:45121 (local address 127.0.0.1:37710)
0109 21:09:14.182486 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.182547 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.125:45121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20250109 21:09:14.183303 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.183111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:45121 (local address 127.0.0.1:37718)
0109 21:09:14.183212 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.183250 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.125:45121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20250109 21:09:14.183411 2676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.125:45121: Network error: Client connection negotiation failed: client connection to 127.12.126.125:45121: connect: Connection refused (error 111)
W20250109 21:09:14.240921 2683 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250109 21:09:14.241231 2683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:09:14.241288 2683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:09:14.241381 2683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:09:14.244459 2683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:09:14.244527 2683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:09:14.244577 2683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:09:14.244608 2683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:09:14.244643 2683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:09:14.248921 2683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:41873,127.12.126.125:45121,127.12.126.124:42171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.125.keytab
--principal=kudu/127.12.126.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.125:45121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.126.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:09:14.250094 2683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:09:14.251013 2683 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250109 21:09:14.251552 2676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } attempt: 2
I20250109 21:09:14.252373 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.252056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:45121 (local address 127.0.0.1:37734)
0109 21:09:14.252219 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.252287 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.125:45121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20250109 21:09:14.253136 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.252919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:45121 (local address 127.0.0.1:37750)
0109 21:09:14.253032 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.253077 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.125:45121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20250109 21:09:14.253260 2676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.125:45121: Network error: Client connection negotiation failed: client connection to 127.12.126.125:45121: connect: Connection refused (error 111)
W20250109 21:09:14.258078 2691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:14.258203 2688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:14.258275 2689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:09:14.258541 2683 server_base.cc:1034] running on GCE node
I20250109 21:09:14.259048 2683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:09:14.259917 2683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:09:14.261072 2683 hybrid_clock.cc:648] HybridClock initialized: now 1736456954261061 us; error 23 us; skew 500 ppm
Jan 09 21:09:14 dist-test-slave-g0zb krb5kdc[2595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456954, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:09:14.263583 2683 init.cc:377] Logged in from keytab as kudu/127.12.126.125@KRBTEST.COM (short username kudu)
I20250109 21:09:14.264442 2683 webserver.cc:458] Webserver started at http://127.12.126.125:42453/ using document root <none> and password file <none>
I20250109 21:09:14.264959 2683 fs_manager.cc:362] Metadata directory not provided
I20250109 21:09:14.265009 2683 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:09:14.265204 2683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:09:14.266981 2683 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-1/data/instance:
uuid: "f010884ff93c45a6a0a92194da28fb1c"
format_stamp: "Formatted at 2025-01-09 21:09:14 on dist-test-slave-g0zb"
server_key: "8357923c5266f6cda7b216bd42c0775f"
server_key_iv: "0b7c4ab68de0fa02cbaa0c7e375497f2"
server_key_version: "encryptionkey@0"
I20250109 21:09:14.267426 2683 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-1/wal/instance:
uuid: "f010884ff93c45a6a0a92194da28fb1c"
format_stamp: "Formatted at 2025-01-09 21:09:14 on dist-test-slave-g0zb"
server_key: "8357923c5266f6cda7b216bd42c0775f"
server_key_iv: "0b7c4ab68de0fa02cbaa0c7e375497f2"
server_key_version: "encryptionkey@0"
I20250109 21:09:14.270746 2683 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250109 21:09:14.272951 2698 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:09:14.273990 2683 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:09:14.274096 2683 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-1/wal
uuid: "f010884ff93c45a6a0a92194da28fb1c"
format_stamp: "Formatted at 2025-01-09 21:09:14 on dist-test-slave-g0zb"
server_key: "8357923c5266f6cda7b216bd42c0775f"
server_key_iv: "0b7c4ab68de0fa02cbaa0c7e375497f2"
server_key_version: "encryptionkey@0"
I20250109 21:09:14.274199 2683 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:09:14.292618 2683 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:09:14.295372 2683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:09:14.295524 2683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:09:14.301980 2683 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.125:45121
I20250109 21:09:14.301985 2751 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.125:45121 every 8 connection(s)
I20250109 21:09:14.303037 2683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-1/data/info.pb
I20250109 21:09:14.305866 2752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:09:14.309123 2752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } has no permanent_uuid. Determining permanent_uuid...
I20250109 21:09:14.312893 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 2683
I20250109 21:09:14.313027 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-1/wal/instance
I20250109 21:09:14.313264 12793 external_mini_cluster.cc:1453] Setting key a97db816784cdce78d983c9768ea5d75
Jan 09 21:09:14 dist-test-slave-g0zb krb5kdc[2595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456954, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.125@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.126.124@KRBTEST.COM; defaulting to no policy
I20250109 21:09:14.321725 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.310977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58372 (local address 127.12.126.126:41873)
0109 21:09:14.311121 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.311126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.311852 (+ 726us) server_negotiation.cc:407] Connection header received
0109 21:09:14.312705 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.312708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.312772 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.312847 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.314205 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.314635 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.315251 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.315411 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.317795 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:14.317815 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:14.317817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:14.317846 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:14.319558 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.320118 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.320123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.320125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.320178 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.320458 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.320462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.320464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.320700 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:14.320806 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:14.321324 (+ 518us) server_negotiation.cc:299] Negotiation successful
0109 21:09:14.321548 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.124@KRBTEST.COM" created.
I20250109 21:09:14.322369 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.311241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:41873 (local address 127.0.0.1:58372)
0109 21:09:14.311711 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.311748 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0109 21:09:14.312480 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:09:14.313031 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:09:14.313044 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:09:14.313476 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:09:14.314050 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:14.314065 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.314744 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.314747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:09:14.315129 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:14.315138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.315345 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.315952 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:09:14.315977 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:09:14.317630 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:09:14.319717 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:14.319724 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:14.319734 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:14.320024 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:14.320259 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:14.320262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:14.320264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:14.320355 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:14.320804 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:09:14.320811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:09:14.321075 (+ 264us) client_negotiation.cc:769] Sending connection context
0109 21:09:14.321268 (+ 193us) client_negotiation.cc:240] Negotiation successful
0109 21:09:14.321565 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":141,"threads_started":1}
I20250109 21:09:14.323802 2752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } has no permanent_uuid. Determining permanent_uuid...
Jan 09 21:09:14 dist-test-slave-g0zb krb5kdc[2595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456954, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.125@KRBTEST.COM for kudu/127.12.126.125@KRBTEST.COM
I20250109 21:09:14.332669 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.324297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:45121 (local address 127.0.0.1:37764)
0109 21:09:14.324394 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.324410 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:09:14.324489 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:09:14.325018 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:09:14.325024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:09:14.325223 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:09:14.325483 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:14.325495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.326441 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.326446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:09:14.327128 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:14.327143 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.327299 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.327850 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:09:14.327871 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:09:14.329189 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:09:14.331250 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:14.331254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:14.331256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:14.331412 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:14.331713 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:14.331716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:14.331718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:14.331754 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:14.332143 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:09:14.332146 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:09:14.332217 (+ 71us) client_negotiation.cc:769] Sending connection context
0109 21:09:14.332313 (+ 96us) client_negotiation.cc:240] Negotiation successful
0109 21:09:14.332467 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20250109 21:09:14.332682 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.324383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.12.126.125:45121)
0109 21:09:14.324676 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.324681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.324700 (+ 19us) server_negotiation.cc:407] Connection header received
0109 21:09:14.324741 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.324745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.324877 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.325001 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.325601 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.326326 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.327267 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.327434 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.329300 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:14.329321 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:14.329332 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:14.329353 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:14.331130 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.331561 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.331569 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.331571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.331637 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.331819 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.331822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.331823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.332053 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:14.332183 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:14.332409 (+ 226us) server_negotiation.cc:299] Negotiation successful
0109 21:09:14.332550 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":121,"threads_started":1}
I20250109 21:09:14.334496 2752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } has no permanent_uuid. Determining permanent_uuid...
I20250109 21:09:14.335253 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.334978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:42171 (local address 127.0.0.1:36396)
0109 21:09:14.335118 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.335186 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20250109 21:09:14.335414 2700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
I20250109 21:09:14.336181 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.335982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:42171 (local address 127.0.0.1:36406)
0109 21:09:14.336065 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.336124 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20250109 21:09:14.336293 2752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.124:42171: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.124.keytab.
Entry for principal kudu/127.12.126.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.124.keytab.
I20250109 21:09:14.337857 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.124:42171
--webserver_interface=127.12.126.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.124.keytab
--principal=kudu/127.12.126.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:41873,127.12.126.125:45121,127.12.126.124:42171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:09:14.344393 2676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } attempt: 3
Jan 09 21:09:14 dist-test-slave-g0zb krb5kdc[2595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456954, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for kudu/127.12.126.125@KRBTEST.COM
I20250109 21:09:14.352201 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.344952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:45121 (local address 127.0.0.1:37770)
0109 21:09:14.345107 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.345122 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0109 21:09:14.345227 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:09:14.345453 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:09:14.345455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:09:14.345616 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:09:14.345793 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:14.345800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.346436 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.346439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:09:14.346766 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:14.346773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.346925 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.347406 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:09:14.347420 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:09:14.349133 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:09:14.351060 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:14.351063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:14.351065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:14.351230 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:14.351446 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:14.351447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:14.351449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:14.351485 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:14.351809 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:09:14.351811 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:09:14.351897 (+ 86us) client_negotiation.cc:769] Sending connection context
0109 21:09:14.351972 (+ 75us) client_negotiation.cc:240] Negotiation successful
0109 21:09:14.352077 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20250109 21:09:14.352283 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.344986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37770 (local address 127.12.126.125:45121)
0109 21:09:14.345105 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.345109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.345179 (+ 70us) server_negotiation.cc:407] Connection header received
0109 21:09:14.345313 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.345316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.345356 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.345415 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.345907 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.346337 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.346922 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.347160 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.349261 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:14.349280 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:14.349283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:14.349317 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:14.350946 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.351319 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.351322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.351324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.351373 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.351546 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.351548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.351550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.351739 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:14.351828 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:14.352026 (+ 198us) server_negotiation.cc:299] Negotiation successful
0109 21:09:14.352140 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250109 21:09:14.353343 2676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } has no permanent_uuid. Determining permanent_uuid...
I20250109 21:09:14.354082 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.353845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:42171 (local address 127.0.0.1:36414)
0109 21:09:14.353968 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.354018 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20250109 21:09:14.354707 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.354493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:42171 (local address 127.0.0.1:36420)
0109 21:09:14.354613 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.354656 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20250109 21:09:14.354874 2676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.124:42171: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
I20250109 21:09:14.385510 2752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } attempt: 1
I20250109 21:09:14.386061 2676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } attempt: 1
I20250109 21:09:14.386500 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.386091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:42171 (local address 127.0.0.1:36424)
0109 21:09:14.386290 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.386370 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20250109 21:09:14.386890 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.386471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:42171 (local address 127.0.0.1:36436)
0109 21:09:14.386716 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.386776 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20250109 21:09:14.387251 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.387039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:42171 (local address 127.0.0.1:36438)
0109 21:09:14.387135 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.387184 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20250109 21:09:14.387385 2752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.124:42171: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
I20250109 21:09:14.387670 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.387426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:42171 (local address 127.0.0.1:36442)
0109 21:09:14.387563 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.387605 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20250109 21:09:14.387820 2676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.124:42171: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
W20250109 21:09:14.445410 2759 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250109 21:09:14.445700 2759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:09:14.445747 2759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:09:14.445832 2759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:09:14.449013 2759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:09:14.449095 2759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:09:14.449131 2759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:09:14.449164 2759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:09:14.449203 2759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:09:14.453552 2759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:41873,127.12.126.125:45121,127.12.126.124:42171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.124.keytab
--principal=kudu/127.12.126.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.124:42171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.126.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:09:14.454670 2759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:09:14.455593 2759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250109 21:09:14.455529 2752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } attempt: 2
I20250109 21:09:14.456319 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.456021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:42171 (local address 127.0.0.1:36446)
0109 21:09:14.456157 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.456217 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20250109 21:09:14.456960 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.456745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:42171 (local address 127.0.0.1:36462)
0109 21:09:14.456865 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.456905 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20250109 21:09:14.457100 2752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.124:42171: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
W20250109 21:09:14.462554 2767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:14.462749 2765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:14.462980 2764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:09:14.462955 2676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } attempt: 2
I20250109 21:09:14.463239 2759 server_base.cc:1034] running on GCE node
I20250109 21:09:14.463696 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.463459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:42171 (local address 127.0.0.1:36470)
0109 21:09:14.463582 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.463626 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20250109 21:09:14.463763 2759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:09:14.464392 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.464170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:42171 (local address 127.0.0.1:36480)
0109 21:09:14.464280 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.464337 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20250109 21:09:14.464525 2676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.124:42171: Network error: Client connection negotiation failed: client connection to 127.12.126.124:42171: connect: Connection refused (error 111)
I20250109 21:09:14.464771 2759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:09:14.465916 2759 hybrid_clock.cc:648] HybridClock initialized: now 1736456954465901 us; error 29 us; skew 500 ppm
Jan 09 21:09:14 dist-test-slave-g0zb krb5kdc[2595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456954, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:09:14.468518 2759 init.cc:377] Logged in from keytab as kudu/127.12.126.124@KRBTEST.COM (short username kudu)
I20250109 21:09:14.469441 2759 webserver.cc:458] Webserver started at http://127.12.126.124:40551/ using document root <none> and password file <none>
I20250109 21:09:14.469987 2759 fs_manager.cc:362] Metadata directory not provided
I20250109 21:09:14.470047 2759 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:09:14.470228 2759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:09:14.472115 2759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-2/data/instance:
uuid: "376318453d3046ab95b419bc4feb8859"
format_stamp: "Formatted at 2025-01-09 21:09:14 on dist-test-slave-g0zb"
server_key: "ff05a0f607c17a45b245009c6afb9359"
server_key_iv: "f045b30a81d1e738268f4b26056c4370"
server_key_version: "encryptionkey@0"
I20250109 21:09:14.472630 2759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-2/wal/instance:
uuid: "376318453d3046ab95b419bc4feb8859"
format_stamp: "Formatted at 2025-01-09 21:09:14 on dist-test-slave-g0zb"
server_key: "ff05a0f607c17a45b245009c6afb9359"
server_key_iv: "f045b30a81d1e738268f4b26056c4370"
server_key_version: "encryptionkey@0"
I20250109 21:09:14.475970 2759 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 21:09:14.477998 2774 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:09:14.479101 2759 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 21:09:14.479215 2759 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-2/wal
uuid: "376318453d3046ab95b419bc4feb8859"
format_stamp: "Formatted at 2025-01-09 21:09:14 on dist-test-slave-g0zb"
server_key: "ff05a0f607c17a45b245009c6afb9359"
server_key_iv: "f045b30a81d1e738268f4b26056c4370"
server_key_version: "encryptionkey@0"
I20250109 21:09:14.479305 2759 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:09:14.521914 2759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:09:14.524628 2759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:09:14.524804 2759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:09:14.531275 2759 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.124:42171
I20250109 21:09:14.531286 2827 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.124:42171 every 8 connection(s)
I20250109 21:09:14.532259 2759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-2/data/info.pb
I20250109 21:09:14.533721 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 2759
I20250109 21:09:14.533869 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1736456573526088-12793-0/minicluster-data/master-2/wal/instance
I20250109 21:09:14.534129 12793 external_mini_cluster.cc:1453] Setting key d52f8adc2deb506f986f2ab640d1b973
I20250109 21:09:14.535609 2828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:09:14.539541 2828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } has no permanent_uuid. Determining permanent_uuid...
Jan 09 21:09:14 dist-test-slave-g0zb krb5kdc[2595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.124@KRBTEST.COM
Jan 09 21:09:14 dist-test-slave-g0zb krb5kdc[2595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456954, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.124@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:09:14.547329 2831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.535307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.12.126.124:42171)
0109 21:09:14.535707 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.535720 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.535758 (+ 38us) server_negotiation.cc:407] Connection header received
0109 21:09:14.536339 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.536356 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.536720 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.537038 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.538079 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.538891 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.539807 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.540045 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.542870 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:14.542930 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:14.542938 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:14.542974 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:14.544864 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.545265 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.545272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.545277 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.545349 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.545529 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.545535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.545537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.545889 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:14.546060 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:14.546333 (+ 273us) server_negotiation.cc:299] Negotiation successful
0109 21:09:14.546591 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":119,"threads_started":1}
I20250109 21:09:14.548244 2752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } attempt: 3
I20250109 21:09:14.550889 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.541520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58376 (local address 127.12.126.126:41873)
0109 21:09:14.541707 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.541712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.542109 (+ 397us) server_negotiation.cc:407] Connection header received
0109 21:09:14.542283 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.542289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.542352 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.542437 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.543345 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.544000 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.544559 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.544777 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.546645 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:14.546666 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:14.546669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:14.546698 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:14.548982 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.549531 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.549535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.549537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.549607 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.549897 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.549901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.549904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.550095 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:14.550166 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:14.550614 (+ 448us) server_negotiation.cc:299] Negotiation successful
0109 21:09:14.550735 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20250109 21:09:14.551393 2832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.541422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:41873 (local address 127.0.0.1:58376)
0109 21:09:14.541967 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.542014 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0109 21:09:14.542154 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:09:14.542512 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:09:14.542516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:09:14.542964 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:09:14.543214 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:14.543226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.544109 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.544112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:09:14.544451 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:14.544458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.544548 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.545011 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:09:14.545028 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:09:14.546455 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:09:14.549125 (+ 2670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:14.549131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:14.549136 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:14.549389 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:14.549725 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:14.549728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:14.549730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:14.549796 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:14.550247 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:09:14.550253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:09:14.550358 (+ 105us) client_negotiation.cc:769] Sending connection context
0109 21:09:14.551033 (+ 675us) client_negotiation.cc:240] Negotiation successful
0109 21:09:14.551209 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":206,"threads_started":1}
I20250109 21:09:14.552840 2828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } has no permanent_uuid. Determining permanent_uuid...
Jan 09 21:09:14 dist-test-slave-g0zb krb5kdc[2595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456954, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.125@KRBTEST.COM for kudu/127.12.126.124@KRBTEST.COM
Jan 09 21:09:14 dist-test-slave-g0zb krb5kdc[2595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
Jan 09 21:09:14 dist-test-slave-g0zb krb5kdc[2595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456954, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.124@KRBTEST.COM for kudu/127.12.126.125@KRBTEST.COM
I20250109 21:09:14.557323 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.548711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:42171 (local address 127.0.0.1:36492)
0109 21:09:14.548841 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.548858 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0109 21:09:14.548960 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:09:14.549272 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:09:14.549276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:09:14.549469 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:09:14.549728 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:14.549738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.550680 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.550683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:09:14.551247 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:14.551259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.551371 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.551905 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:09:14.551922 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:09:14.553363 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:09:14.555375 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:14.555379 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:14.555381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:14.555591 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:14.556311 (+ 720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:14.556317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:14.556321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:14.556383 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:14.556789 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:09:14.556793 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:09:14.556891 (+ 98us) client_negotiation.cc:769] Sending connection context
0109 21:09:14.556988 (+ 97us) client_negotiation.cc:240] Negotiation successful
0109 21:09:14.557116 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20250109 21:09:14.557916 2831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.548912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.12.126.124:42171)
0109 21:09:14.549045 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.549049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.549060 (+ 11us) server_negotiation.cc:407] Connection header received
0109 21:09:14.549091 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.549093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.549135 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.549839 (+ 704us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.549923 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.550559 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.551360 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.551884 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.553520 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:14.553538 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:14.553541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:14.553570 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:14.555261 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.555698 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.555701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.555703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.555756 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.556484 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.556487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.556489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.556656 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:14.557526 (+ 870us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:14.557609 (+ 83us) server_negotiation.cc:299] Negotiation successful
0109 21:09:14.557724 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250109 21:09:14.559446 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.549743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58378 (local address 127.12.126.126:41873)
0109 21:09:14.550078 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.550081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.550091 (+ 10us) server_negotiation.cc:407] Connection header received
0109 21:09:14.550119 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.550121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.550154 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.550238 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.551084 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.552121 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.553028 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.554335 (+ 1307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.555533 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:14.555555 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:14.555558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:14.555586 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:14.557358 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.558037 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.558040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.558041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.558099 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.558399 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.558402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.558404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.558567 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:14.558760 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:14.559156 (+ 396us) server_negotiation.cc:299] Negotiation successful
0109 21:09:14.559286 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":121,"threads_started":1}
I20250109 21:09:14.560740 2832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.553324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:45121 (local address 127.0.0.1:37772)
0109 21:09:14.553428 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.553442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:09:14.553516 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:09:14.553961 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:09:14.553965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:09:14.554144 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:09:14.554359 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:14.554368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.555354 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.555357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:09:14.555775 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:14.555787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.555899 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.556274 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:09:14.556292 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:09:14.557462 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:09:14.559531 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:14.559535 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:14.559537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:14.559703 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:14.559914 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:14.559917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:14.559919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:14.559960 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:14.560255 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:09:14.560258 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:09:14.560329 (+ 71us) client_negotiation.cc:769] Sending connection context
0109 21:09:14.560441 (+ 112us) client_negotiation.cc:240] Negotiation successful
0109 21:09:14.560566 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20250109 21:09:14.560748 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.553562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37772 (local address 127.12.126.125:45121)
0109 21:09:14.553662 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.553665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.553675 (+ 10us) server_negotiation.cc:407] Connection header received
0109 21:09:14.553722 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.553724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.553757 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.553831 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.554525 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.555232 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.555874 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.556039 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.557592 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:14.557608 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:14.557611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:14.557641 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:14.559432 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.559797 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.559800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.559802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.559843 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.560029 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.560032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.560034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.560175 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:14.560254 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:14.560498 (+ 244us) server_negotiation.cc:299] Negotiation successful
0109 21:09:14.560604 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20250109 21:09:14.560928 2752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c: Bootstrap starting.
I20250109 21:09:14.561475 2828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } has no permanent_uuid. Determining permanent_uuid...
I20250109 21:09:14.563658 2676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } attempt: 3
I20250109 21:09:14.563946 2752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c: Neither blocks nor log segments found. Creating new log.
I20250109 21:09:14.565196 2752 log.cc:826] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c: Log is configured to *not* fsync() on all Append() calls
Jan 09 21:09:14 dist-test-slave-g0zb krb5kdc[2595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456954, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.124@KRBTEST.COM for kudu/127.12.126.124@KRBTEST.COM
I20250109 21:09:14.568377 2752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c: No bootstrap required, opened a new log
Jan 09 21:09:14 dist-test-slave-g0zb krb5kdc[2595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456954, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for kudu/127.12.126.124@KRBTEST.COM
Jan 09 21:09:14 dist-test-slave-g0zb krb5kdc[2595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.125@KRBTEST.COM
I20250109 21:09:14.571594 2752 raft_consensus.cc:357] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } } peers { permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } } peers { permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } }
I20250109 21:09:14.571838 2752 raft_consensus.cc:738] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f010884ff93c45a6a0a92194da28fb1c, State: Initialized, Role: FOLLOWER
I20250109 21:09:14.572203 2752 consensus_queue.cc:260] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } } peers { permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } } peers { permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } }
I20250109 21:09:14.572567 2832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.561940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:42171 (local address 127.0.0.1:36504)
0109 21:09:14.562063 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.562077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:09:14.562184 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:09:14.562448 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:09:14.562452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:09:14.562616 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:09:14.562782 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:14.562788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.563645 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.563649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:09:14.564085 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:14.564096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.564211 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.564707 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:09:14.564721 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:09:14.566192 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:09:14.569912 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:14.569915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:14.569917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:14.570115 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:14.570352 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:14.570356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:14.570359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:14.570409 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:14.572063 (+ 1654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:09:14.572067 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:09:14.572165 (+ 98us) client_negotiation.cc:769] Sending connection context
0109 21:09:14.572263 (+ 98us) client_negotiation.cc:240] Negotiation successful
0109 21:09:14.572383 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20250109 21:09:14.573540 2752 sys_catalog.cc:564] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:09:14.573689 2831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.561985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36504 (local address 127.12.126.124:42171)
0109 21:09:14.562111 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.562117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.562132 (+ 15us) server_negotiation.cc:407] Connection header received
0109 21:09:14.562276 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.562279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.562341 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.562420 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.562912 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.563513 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.564407 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.564602 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.566307 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:14.566323 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:14.566326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:14.566350 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:14.569782 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.570221 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.570226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.570229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.570269 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.570897 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.570904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.570906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.571555 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:14.573321 (+ 1766us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:14.573389 (+ 68us) server_negotiation.cc:299] Negotiation successful
0109 21:09:14.573498 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250109 21:09:14.574120 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.563991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:42171 (local address 127.0.0.1:36516)
0109 21:09:14.564230 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:14.564244 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:09:14.564322 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:09:14.565394 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:09:14.565397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:09:14.565572 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:09:14.565817 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:14.565827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.566733 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.566738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:09:14.567302 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:14.567317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.567434 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.567981 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:09:14.568004 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:09:14.570116 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:09:14.572486 (+ 2370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:14.572492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:14.572494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:14.572702 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:14.573161 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:14.573164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:14.573166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:14.573220 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:14.573600 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:09:14.573603 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:09:14.573707 (+ 104us) client_negotiation.cc:769] Sending connection context
0109 21:09:14.573812 (+ 105us) client_negotiation.cc:240] Negotiation successful
0109 21:09:14.573953 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20250109 21:09:14.574579 2834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.564428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.12.126.124:42171)
0109 21:09:14.565057 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.565063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.565077 (+ 14us) server_negotiation.cc:407] Connection header received
0109 21:09:14.565111 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.565116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.565183 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.565995 (+ 812us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.566101 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.566640 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.567640 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.567852 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.570239 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:14.570257 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:14.570260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:14.570287 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:14.572185 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.573009 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.573012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.573013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.573061 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.573309 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.573312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.573314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.573476 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:14.573557 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:14.574321 (+ 764us) server_negotiation.cc:299] Negotiation successful
0109 21:09:14.574426 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":96,"threads_started":1}
I20250109 21:09:14.574869 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.564167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37788 (local address 127.12.126.125:45121)
0109 21:09:14.564279 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.564282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.564804 (+ 522us) server_negotiation.cc:407] Connection header received
0109 21:09:14.564905 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.564907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.564956 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.565029 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.565961 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.566479 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.567701 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.567869 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.570867 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:14.570902 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:14.570906 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:14.570931 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:14.572566 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.573014 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.573017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.573019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.573066 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.573437 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.573441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.573443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.573605 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:14.573776 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:14.574573 (+ 797us) server_negotiation.cc:299] Negotiation successful
0109 21:09:14.574697 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250109 21:09:14.577174 2828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859: Bootstrap starting.
I20250109 21:09:14.577216 2676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e: Bootstrap starting.
W20250109 21:09:14.579749 2847 catalog_manager.cc:1559] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:09:14.579849 2847 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 21:09:14.577574 2683 master_runner.cc:186] Error getting master registration for 127.12.126.126:41873: OK, instance_id { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" instance_seqno: 1736456954101409 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250109 21:09:14.580210 2683 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 21:09:14.580693 2828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859: Neither blocks nor log segments found. Creating new log.
I20250109 21:09:14.581275 2836 sys_catalog.cc:455] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } } peers { permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } } peers { permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } } }
I20250109 21:09:14.581401 2836 sys_catalog.cc:458] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:09:14.581830 2676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e: Neither blocks nor log segments found. Creating new log.
I20250109 21:09:14.581845 2828 log.cc:826] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859: Log is configured to *not* fsync() on all Append() calls
I20250109 21:09:14.582135 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:09:14.582671 2676 log.cc:826] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e: Log is configured to *not* fsync() on all Append() calls
I20250109 21:09:14.584261 2828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859: No bootstrap required, opened a new log
I20250109 21:09:14.583491 2650 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "885708b5be4045da8a4eb11deefb566e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58378
I20250109 21:09:14.584654 2676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e: No bootstrap required, opened a new log
I20250109 21:09:14.587138 2676 raft_consensus.cc:357] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } } peers { permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } } peers { permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } }
I20250109 21:09:14.587370 2676 raft_consensus.cc:738] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885708b5be4045da8a4eb11deefb566e, State: Initialized, Role: FOLLOWER
I20250109 21:09:14.587576 2828 raft_consensus.cc:357] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } } peers { permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } } peers { permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } }
I20250109 21:09:14.587769 2828 raft_consensus.cc:738] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376318453d3046ab95b419bc4feb8859, State: Initialized, Role: FOLLOWER
I20250109 21:09:14.587734 2676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } } peers { permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } } peers { permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } }
I20250109 21:09:14.588124 2828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } } peers { permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } } peers { permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } }
I20250109 21:09:14.588644 2852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } } peers { permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } } peers { permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } } }
I20250109 21:09:14.588917 2852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:09:14.589085 2676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:09:14.589184 2650 raft_consensus.cc:491] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250109 21:09:14.589284 2828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:09:14.589285 2650 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:09:14.590677 2650 raft_consensus.cc:513] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } } peers { permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } } peers { permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } }
I20250109 21:09:14.591794 2650 leader_election.cc:290] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [CANDIDATE]: Term 1 election: Requested vote from peers f010884ff93c45a6a0a92194da28fb1c (127.12.126.125:45121), 376318453d3046ab95b419bc4feb8859 (127.12.126.124:42171)
I20250109 21:09:14.592417 2725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "885708b5be4045da8a4eb11deefb566e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f010884ff93c45a6a0a92194da28fb1c"
I20250109 21:09:14.592670 2725 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:09:14.593190 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "885708b5be4045da8a4eb11deefb566e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "376318453d3046ab95b419bc4feb8859"
I20250109 21:09:14.593390 2802 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:09:14.593997 2725 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 885708b5be4045da8a4eb11deefb566e in term 1.
I20250109 21:09:14.594491 2627 leader_election.cc:304] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 885708b5be4045da8a4eb11deefb566e, f010884ff93c45a6a0a92194da28fb1c; no voters:
I20250109 21:09:14.594507 2802 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 885708b5be4045da8a4eb11deefb566e in term 1.
I20250109 21:09:14.595127 2881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } } peers { permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } } peers { permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } } }
I20250109 21:09:14.595261 2881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:09:14.595333 2881 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:09:14.595391 2881 raft_consensus.cc:695] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [term 1 LEADER]: Becoming Leader. State: Replica: 885708b5be4045da8a4eb11deefb566e, State: Running, Role: LEADER
I20250109 21:09:14.595692 2881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } } peers { permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } } peers { permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } }
W20250109 21:09:14.599069 2629 tablet.cc:2367] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:09:14.600086 2881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 885708b5be4045da8a4eb11deefb566e. Latest consensus state: current_term: 1 leader_uuid: "885708b5be4045da8a4eb11deefb566e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } } peers { permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } } peers { permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } } }
I20250109 21:09:14.600199 2881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [sys.catalog]: This master's current role is: LEADER
W20250109 21:09:14.603487 2872 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:09:14.603574 2872 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 21:09:14.603950 2759 master_runner.cc:263] No leader master found from master 376318453d3046ab95b419bc4feb8859
I20250109 21:09:14.604013 2759 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 21:09:14.604048 2874 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:09:14.604126 2874 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 21:09:14.615084 2831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.604525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36520 (local address 127.12.126.124:42171)
0109 21:09:14.604647 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.604652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.605512 (+ 860us) server_negotiation.cc:407] Connection header received
0109 21:09:14.605731 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.605735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.605782 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.605835 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.607113 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.607845 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.608688 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.608892 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.610714 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:14.610735 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:14.610739 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:14.610769 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:14.612801 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.613804 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.613807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.613809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.613859 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.614098 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.614101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.614102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.614277 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:14.614395 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:14.614746 (+ 351us) server_negotiation.cc:299] Negotiation successful
0109 21:09:14.614905 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20250109 21:09:14.622428 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.605020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58386 (local address 127.12.126.126:41873)
0109 21:09:14.605128 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.605132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.609904 (+ 4772us) server_negotiation.cc:407] Connection header received
0109 21:09:14.611559 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.611563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.611629 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.612005 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.614925 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.615638 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.616432 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.617007 (+ 575us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.618581 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:14.618603 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:14.618606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:14.618638 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:14.620552 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.621082 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.621088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.621091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.621149 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.621399 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.621402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.621405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.621585 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:14.621726 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:14.622116 (+ 390us) server_negotiation.cc:299] Negotiation successful
0109 21:09:14.622254 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20250109 21:09:14.627059 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:14.605545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.12.126.125:45121)
0109 21:09:14.605665 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.605668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.610120 (+ 4452us) server_negotiation.cc:407] Connection header received
0109 21:09:14.610254 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.610258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.610313 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.610398 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.611864 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.612626 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:14.615922 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:14.616142 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:14.622951 (+ 6809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:14.622976 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:14.622980 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:14.623016 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:14.625356 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.625844 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.625848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.625850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.625906 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:14.626146 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:14.626149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:14.626151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:14.626322 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:14.626421 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:14.626744 (+ 323us) server_negotiation.cc:299] Negotiation successful
0109 21:09:14.626899 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20250109 21:09:14.627499 2883 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:09:14.628544 2883 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 21:09:14.634465 2802 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859 [term 1 FOLLOWER]: Refusing update from remote peer 885708b5be4045da8a4eb11deefb566e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:09:14.635004 2724 raft_consensus.cc:1270] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c [term 1 FOLLOWER]: Refusing update from remote peer 885708b5be4045da8a4eb11deefb566e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:09:14.635638 2881 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [LEADER]: Connected to new peer: Peer: permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:09:14.635936 2851 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:09:14.639962 2836 sys_catalog.cc:455] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 885708b5be4045da8a4eb11deefb566e. Latest consensus state: current_term: 1 leader_uuid: "885708b5be4045da8a4eb11deefb566e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } } peers { permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } } peers { permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } } }
I20250109 21:09:14.640094 2836 sys_catalog.cc:458] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:09:14.645674 2852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 885708b5be4045da8a4eb11deefb566e. Latest consensus state: current_term: 1 leader_uuid: "885708b5be4045da8a4eb11deefb566e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } } peers { permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } } peers { permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } } }
I20250109 21:09:14.645818 2852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:09:14.655670 2881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "885708b5be4045da8a4eb11deefb566e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } } peers { permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } } peers { permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } } }
I20250109 21:09:14.655838 2881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:14.655944 2888 mvcc.cc:204] Tried to move back new op lower bound from 7112527686173540352 to 7112527686028906496. Current Snapshot: MvccSnapshot[applied={T|T < 7112527686173540352}]
I20250109 21:09:14.657392 2836 sys_catalog.cc:455] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "885708b5be4045da8a4eb11deefb566e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } } peers { permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } } peers { permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } } }
I20250109 21:09:14.657521 2836 sys_catalog.cc:458] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:09:14.661165 2851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "885708b5be4045da8a4eb11deefb566e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } } peers { permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } } peers { permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } } }
I20250109 21:09:14.661322 2851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:14.662528 2852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "885708b5be4045da8a4eb11deefb566e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885708b5be4045da8a4eb11deefb566e" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 41873 } } peers { permanent_uuid: "f010884ff93c45a6a0a92194da28fb1c" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 45121 } } peers { permanent_uuid: "376318453d3046ab95b419bc4feb8859" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 42171 } } }
I20250109 21:09:14.662668 2852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:09:14.663749 2889 mvcc.cc:204] Tried to move back new op lower bound from 7112527686173540352 to 7112527686028906496. Current Snapshot: MvccSnapshot[applied={T|T < 7112527686173540352}]
I20250109 21:09:14.663928 2891 mvcc.cc:204] Tried to move back new op lower bound from 7112527686173540352 to 7112527686028906496. Current Snapshot: MvccSnapshot[applied={T|T < 7112527686173540352}]
I20250109 21:09:14.669135 2883 catalog_manager.cc:1348] Generated new cluster ID: 6d3a78f0ce874980877ae48831a3bd34
I20250109 21:09:14.669220 2883 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:09:14.688060 2883 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:09:14.689411 2883 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:09:14.708508 2883 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 885708b5be4045da8a4eb11deefb566e: Generated new TSK 0
I20250109 21:09:14.709457 2883 catalog_manager.cc:1515] Initializing in-progress tserver states...
W20250109 21:09:14.739959 2831 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:14.736972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36532 (local address 127.12.126.124:42171)
0109 21:09:14.737122 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.737127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.738792 (+ 1665us) server_negotiation.cc:407] Connection header received
0109 21:09:14.739038 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.739042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.739098 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.739161 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.739807 (+ 646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36532: BlockingRecv error: recv got EOF from 127.0.0.1:36532 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20250109 21:09:14.741289 2833 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:14.737316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58388 (local address 127.12.126.126:41873)
0109 21:09:14.737434 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.737438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.740439 (+ 3001us) server_negotiation.cc:407] Connection header received
0109 21:09:14.740620 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.740624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.740677 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.740730 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.741177 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58388: BlockingRecv error: recv got EOF from 127.0.0.1:58388 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20250109 21:09:14.742579 2757 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:14.738193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.12.126.125:45121)
0109 21:09:14.738314 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.738318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.741772 (+ 3454us) server_negotiation.cc:407] Connection header received
0109 21:09:14.741938 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.741941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.741990 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.742041 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.742464 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37812: BlockingRecv error: recv got EOF from 127.0.0.1:37812 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20250109 21:09:14.746556 2757 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:14.745277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.12.126.125:45121)
0109 21:09:14.745413 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.745417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.745638 (+ 221us) server_negotiation.cc:407] Connection header received
0109 21:09:14.745864 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.745867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.745942 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.745996 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.746460 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37822: BlockingRecv error: recv got EOF from 127.0.0.1:37822 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20250109 21:09:14.751343 2757 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:14.749932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.12.126.125:45121)
0109 21:09:14.750031 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.750034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.750473 (+ 439us) server_negotiation.cc:407] Connection header received
0109 21:09:14.750661 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.750664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.750714 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.750763 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.751251 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37838: BlockingRecv error: recv got EOF from 127.0.0.1:37838 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20250109 21:09:14.756793 2757 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:14.755360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37854 (local address 127.12.126.125:45121)
0109 21:09:14.755463 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.755466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.755962 (+ 496us) server_negotiation.cc:407] Connection header received
0109 21:09:14.756135 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.756139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.756186 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.756232 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.756696 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37854: BlockingRecv error: recv got EOF from 127.0.0.1:37854 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20250109 21:09:14.764367 2757 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:14.762918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37864 (local address 127.12.126.125:45121)
0109 21:09:14.763048 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.763052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.763485 (+ 433us) server_negotiation.cc:407] Connection header received
0109 21:09:14.763701 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.763704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.763757 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.763808 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.764268 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37864: BlockingRecv error: recv got EOF from 127.0.0.1:37864 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20250109 21:09:14.776494 2757 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:14.774765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37880 (local address 127.12.126.125:45121)
0109 21:09:14.774908 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.774914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.775469 (+ 555us) server_negotiation.cc:407] Connection header received
0109 21:09:14.775693 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.775697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.775754 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.775806 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.776393 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37880: BlockingRecv error: recv got EOF from 127.0.0.1:37880 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20250109 21:09:14.796618 2757 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:14.794894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37892 (local address 127.12.126.125:45121)
0109 21:09:14.795053 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.795058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.795645 (+ 587us) server_negotiation.cc:407] Connection header received
0109 21:09:14.795862 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.795866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.795924 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.795976 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.796499 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37892: BlockingRecv error: recv got EOF from 127.0.0.1:37892 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20250109 21:09:14.832756 2757 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:14.831045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37902 (local address 127.12.126.125:45121)
0109 21:09:14.831181 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.831185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.831793 (+ 608us) server_negotiation.cc:407] Connection header received
0109 21:09:14.831984 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.831987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.832040 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.832090 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.832636 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37902: BlockingRecv error: recv got EOF from 127.0.0.1:37902 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20250109 21:09:14.899951 2757 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:14.898941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37918 (local address 127.12.126.125:45121)
0109 21:09:14.899101 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:14.899104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:14.899175 (+ 71us) server_negotiation.cc:407] Connection header received
0109 21:09:14.899319 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:14.899322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:14.899374 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:14.899454 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:14.899822 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37918: BlockingRecv error: recv got EOF from 127.0.0.1:37918 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20250109 21:09:15.030995 2757 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:15.029909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37926 (local address 127.12.126.125:45121)
0109 21:09:15.030050 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:15.030053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:15.030154 (+ 101us) server_negotiation.cc:407] Connection header received
0109 21:09:15.030293 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:15.030295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:15.030342 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:15.030432 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:15.030866 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37926: BlockingRecv error: recv got EOF from 127.0.0.1:37926 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20250109 21:09:15.290117 2757 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:15.288988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.12.126.125:45121)
0109 21:09:15.289136 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:15.289141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:15.289270 (+ 129us) server_negotiation.cc:407] Connection header received
0109 21:09:15.289417 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:15.289420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:15.289470 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:15.289543 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:15.290021 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37932: BlockingRecv error: recv got EOF from 127.0.0.1:37932 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20250109 21:09:15.581187 2847 catalog_manager.cc:1260] Loaded cluster ID: 6d3a78f0ce874980877ae48831a3bd34
I20250109 21:09:15.581284 2847 catalog_manager.cc:1553] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c: loading cluster ID for follower catalog manager: success
I20250109 21:09:15.583117 2847 catalog_manager.cc:1575] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c: acquiring CA information for follower catalog manager: success
I20250109 21:09:15.584404 2847 catalog_manager.cc:1603] T 00000000000000000000000000000000 P f010884ff93c45a6a0a92194da28fb1c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 21:09:15.604919 2872 catalog_manager.cc:1260] Loaded cluster ID: 6d3a78f0ce874980877ae48831a3bd34
I20250109 21:09:15.605010 2872 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859: loading cluster ID for follower catalog manager: success
I20250109 21:09:15.606609 2872 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859: acquiring CA information for follower catalog manager: success
I20250109 21:09:15.607758 2872 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 376318453d3046ab95b419bc4feb8859: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 21:09:15.807410 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:15.804178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37944 (local address 127.12.126.125:45121)
0109 21:09:15.804408 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:15.804411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:15.804430 (+ 19us) server_negotiation.cc:407] Connection header received
0109 21:09:15.804572 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:15.804574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:15.804730 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:15.804845 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:09:15.805163 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:15.805648 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:15.806225 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:15.806357 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:15.806446 (+ 89us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:09:15.806966 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:09:15.807088 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:15.807202 (+ 114us) server_negotiation.cc:299] Negotiation successful
0109 21:09:15.807255 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":71,"threads_started":1}
I20250109 21:09:15.813194 2967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:15.810131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.12.126.124:42171)
0109 21:09:15.810352 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:15.810358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:15.810402 (+ 44us) server_negotiation.cc:407] Connection header received
0109 21:09:15.810489 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:15.810492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:15.810615 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:15.810728 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:09:15.811060 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:15.811552 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:15.812171 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:15.812332 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:15.812431 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:09:15.812818 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:09:15.812929 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:15.813006 (+ 77us) server_negotiation.cc:299] Negotiation successful
0109 21:09:15.813059 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":60,"threads_started":1}
I20250109 21:09:15.819437 2967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:15.815962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36562 (local address 127.12.126.124:42171)
0109 21:09:15.816072 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:15.816076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:15.816265 (+ 189us) server_negotiation.cc:407] Connection header received
0109 21:09:15.816463 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:15.816468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:15.816537 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:15.816614 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:09:15.816998 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:15.817566 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:15.818220 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:15.818395 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:15.818437 (+ 42us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:09:15.818505 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:09:15.818608 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:15.819246 (+ 638us) server_negotiation.cc:299] Negotiation successful
0109 21:09:15.819296 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250109 21:09:15.819516 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:15.816116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37956 (local address 127.12.126.125:45121)
0109 21:09:15.816224 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:15.816228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:15.816541 (+ 313us) server_negotiation.cc:407] Connection header received
0109 21:09:15.816709 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:15.816713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:15.816760 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:15.816798 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:09:15.817265 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:15.817979 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:15.818831 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:15.819010 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:15.819057 (+ 47us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:09:15.819128 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:09:15.819205 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:15.819310 (+ 105us) server_negotiation.cc:299] Negotiation successful
0109 21:09:15.819364 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20250109 21:09:15.819860 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:15.815866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58392 (local address 127.12.126.126:41873)
0109 21:09:15.816117 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:15.816121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:15.816312 (+ 191us) server_negotiation.cc:407] Connection header received
0109 21:09:15.816427 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:15.816431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:15.816560 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:15.816695 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:09:15.817134 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:15.817936 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:15.818617 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:15.818859 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:15.818965 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:09:15.819298 (+ 333us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:09:15.819380 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:15.819655 (+ 275us) server_negotiation.cc:299] Negotiation successful
0109 21:09:15.819724 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":63,"threads_started":1}
I20250109 21:09:15.828212 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 2608
I20250109 21:09:15.834676 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 2683
I20250109 21:09:15.840478 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 2759
2025-01-09T21:09:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2992 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 09 21:09:15 dist-test-slave-g0zb krb5kdc[2978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 09 21:09:15 dist-test-slave-g0zb krb5kdc[2978](info): set up 2 sockets
Jan 09 21:09:15 dist-test-slave-g0zb krb5kdc[2978](info): commencing operation
krb5kdc: starting...
W20250109 21:09:16.827510 12793 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/test-user.keytab.
Jan 09 21:09:16 dist-test-slave-g0zb krb5kdc[2978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-01-09T21:09:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T21:09:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.126.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
Entry for principal kudu/127.12.126.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab.
I20250109 21:09:16.934203 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:42939
--webserver_interface=127.12.126.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:38149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:42939,127.12.126.125:35623,127.12.126.124:43459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250109 21:09:17.036582 2991 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250109 21:09:17.036820 2991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:09:17.036846 2991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:09:17.036919 2991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:09:17.039891 2991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:09:17.039944 2991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:09:17.039965 2991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:09:17.039999 2991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:09:17.040035 2991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:09:17.044063 2991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:38149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:42939,127.12.126.125:35623,127.12.126.124:43459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.126.keytab
--principal=kudu/127.12.126.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.126:42939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.126.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:09:17.045043 2991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:09:17.045819 2991 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:09:17.052377 2996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:17.052381 2999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:17.052644 2997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:09:17.052886 2991 server_base.cc:1034] running on GCE node
I20250109 21:09:17.053268 2991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:09:17.054033 2991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:09:17.055182 2991 hybrid_clock.cc:648] HybridClock initialized: now 1736456957055174 us; error 23 us; skew 500 ppm
Jan 09 21:09:17 dist-test-slave-g0zb krb5kdc[2978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:09:17.057386 2991 init.cc:377] Logged in from keytab as kudu/127.12.126.126@KRBTEST.COM (short username kudu)
I20250109 21:09:17.058260 2991 webserver.cc:458] Webserver started at http://127.12.126.126:44701/ using document root <none> and password file <none>
I20250109 21:09:17.058739 2991 fs_manager.cc:362] Metadata directory not provided
I20250109 21:09:17.058790 2991 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:09:17.058982 2991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:09:17.060586 2991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-0/data/instance:
uuid: "2dd663a423654e7eafac4339d4c8bc77"
format_stamp: "Formatted at 2025-01-09 21:09:17 on dist-test-slave-g0zb"
server_key: "8e500a2cf15bceb38fdc419640d7ded7"
server_key_iv: "d7359980058d08bfbe3cccd689a139d8"
server_key_version: "encryptionkey@0"
I20250109 21:09:17.061043 2991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-0/wal/instance:
uuid: "2dd663a423654e7eafac4339d4c8bc77"
format_stamp: "Formatted at 2025-01-09 21:09:17 on dist-test-slave-g0zb"
server_key: "8e500a2cf15bceb38fdc419640d7ded7"
server_key_iv: "d7359980058d08bfbe3cccd689a139d8"
server_key_version: "encryptionkey@0"
I20250109 21:09:17.064229 2991 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20250109 21:09:17.066073 3006 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:09:17.066968 2991 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:09:17.067060 2991 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-0/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-0/wal
uuid: "2dd663a423654e7eafac4339d4c8bc77"
format_stamp: "Formatted at 2025-01-09 21:09:17 on dist-test-slave-g0zb"
server_key: "8e500a2cf15bceb38fdc419640d7ded7"
server_key_iv: "d7359980058d08bfbe3cccd689a139d8"
server_key_version: "encryptionkey@0"
I20250109 21:09:17.067133 2991 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:09:17.089268 2991 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:09:17.091935 2991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:09:17.092109 2991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:09:17.098423 2991 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.126:42939
I20250109 21:09:17.098438 3059 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.126:42939 every 8 connection(s)
I20250109 21:09:17.099380 2991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-0/data/info.pb
I20250109 21:09:17.100054 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 2991
I20250109 21:09:17.100162 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-0/wal/instance
I20250109 21:09:17.100386 12793 external_mini_cluster.cc:1453] Setting key a47a2006db71e499a5f66bbc6afdf4fd
I20250109 21:09:17.102195 3060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:09:17.105396 3060 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.126.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.125@KRBTEST.COM" created.
Jan 09 21:09:17 dist-test-slave-g0zb krb5kdc[2978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:09:17.117605 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.107292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42939 (local address 127.0.0.1:48436)
0109 21:09:17.107889 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.107926 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0109 21:09:17.108600 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:09:17.109261 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:09:17.109265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:09:17.109596 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:09:17.109889 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:17.109901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.110999 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.111003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:09:17.111340 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:17.111346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.111541 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.112224 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:09:17.112245 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:09:17.113522 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:09:17.115308 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:17.115314 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:17.115321 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:17.115498 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:17.115712 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:17.115714 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:17.115716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:17.115773 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:17.116309 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:09:17.116314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:09:17.116424 (+ 110us) client_negotiation.cc:769] Sending connection context
0109 21:09:17.116570 (+ 146us) client_negotiation.cc:240] Negotiation successful
0109 21:09:17.116907 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":96,"spinlock_wait_cycles":54528,"thread_start_us":138,"threads_started":1}
I20250109 21:09:17.117605 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.107292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48436 (local address 127.12.126.126:42939)
0109 21:09:17.107843 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.107851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.108018 (+ 167us) server_negotiation.cc:407] Connection header received
0109 21:09:17.108877 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.108881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.109070 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.109212 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.110015 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.110877 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.111436 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.111619 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.113633 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:17.113649 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:17.113658 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:17.113677 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:17.115221 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.115585 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.115589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.115590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.115640 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.115840 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.115843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.115844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.116213 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:17.116333 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:17.116677 (+ 344us) server_negotiation.cc:299] Negotiation successful
0109 21:09:17.116834 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":142,"threads_started":1}
I20250109 21:09:17.119612 3060 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } has no permanent_uuid. Determining permanent_uuid...
I20250109 21:09:17.120232 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.120012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:35623 (local address 127.0.0.1:40204)
0109 21:09:17.120115 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.120173 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.125:35623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20250109 21:09:17.120398 3008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.126.125:35623: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.125.keytab.
Entry for principal kudu/127.12.126.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.125.keytab.
I20250109 21:09:17.121191 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.120993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:35623 (local address 127.0.0.1:40212)
0109 21:09:17.121097 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.121136 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.125:35623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20250109 21:09:17.121300 3060 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.125:35623: Network error: Client connection negotiation failed: client connection to 127.12.126.125:35623: connect: Connection refused (error 111)
I20250109 21:09:17.121851 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.125:35623
--webserver_interface=127.12.126.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.125.keytab
--principal=kudu/127.12.126.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:38149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:42939,127.12.126.125:35623,127.12.126.124:43459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:09:17.170603 3060 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } attempt: 1
I20250109 21:09:17.171510 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.171215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:35623 (local address 127.0.0.1:40226)
0109 21:09:17.171370 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.171431 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.125:35623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20250109 21:09:17.172200 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.172023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:35623 (local address 127.0.0.1:40228)
0109 21:09:17.172109 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.172149 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.125:35623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20250109 21:09:17.172310 3060 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.125:35623: Network error: Client connection negotiation failed: client connection to 127.12.126.125:35623: connect: Connection refused (error 111)
W20250109 21:09:17.224368 3067 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250109 21:09:17.224614 3067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:09:17.224642 3067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:09:17.224689 3067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:09:17.227695 3067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:09:17.227749 3067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:09:17.227771 3067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:09:17.227792 3067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:09:17.227826 3067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:09:17.231761 3067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:38149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:42939,127.12.126.125:35623,127.12.126.124:43459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.125.keytab
--principal=kudu/127.12.126.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.125:35623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.126.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:09:17.232756 3067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:09:17.233549 3067 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:09:17.240137 3075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:17.240191 3072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:17.240396 3073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:09:17.240453 3060 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } attempt: 2
I20250109 21:09:17.240666 3067 server_base.cc:1034] running on GCE node
I20250109 21:09:17.241133 3067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:09:17.241185 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.240903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:35623 (local address 127.0.0.1:40234)
0109 21:09:17.241062 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.241113 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.125:35623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20250109 21:09:17.241858 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.241653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:35623 (local address 127.0.0.1:40250)
0109 21:09:17.241759 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.241803 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.125:35623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20250109 21:09:17.241983 3060 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.125:35623: Network error: Client connection negotiation failed: client connection to 127.12.126.125:35623: connect: Connection refused (error 111)
I20250109 21:09:17.242039 3067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:09:17.243186 3067 hybrid_clock.cc:648] HybridClock initialized: now 1736456957243168 us; error 29 us; skew 500 ppm
Jan 09 21:09:17 dist-test-slave-g0zb krb5kdc[2978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:09:17.245608 3067 init.cc:377] Logged in from keytab as kudu/127.12.126.125@KRBTEST.COM (short username kudu)
I20250109 21:09:17.246505 3067 webserver.cc:458] Webserver started at http://127.12.126.125:33709/ using document root <none> and password file <none>
I20250109 21:09:17.247058 3067 fs_manager.cc:362] Metadata directory not provided
I20250109 21:09:17.247112 3067 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:09:17.247299 3067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:09:17.249033 3067 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-1/data/instance:
uuid: "75bbfd78c32f40cab6bca49dfd72f7df"
format_stamp: "Formatted at 2025-01-09 21:09:17 on dist-test-slave-g0zb"
server_key: "445105856e444b2ecdd004d0ed76abdf"
server_key_iv: "f12cf4d8364699fc5b41954462656e9f"
server_key_version: "encryptionkey@0"
I20250109 21:09:17.249500 3067 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-1/wal/instance:
uuid: "75bbfd78c32f40cab6bca49dfd72f7df"
format_stamp: "Formatted at 2025-01-09 21:09:17 on dist-test-slave-g0zb"
server_key: "445105856e444b2ecdd004d0ed76abdf"
server_key_iv: "f12cf4d8364699fc5b41954462656e9f"
server_key_version: "encryptionkey@0"
I20250109 21:09:17.252789 3067 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 21:09:17.254755 3082 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:09:17.255738 3067 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250109 21:09:17.255833 3067 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-1/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-1/wal
uuid: "75bbfd78c32f40cab6bca49dfd72f7df"
format_stamp: "Formatted at 2025-01-09 21:09:17 on dist-test-slave-g0zb"
server_key: "445105856e444b2ecdd004d0ed76abdf"
server_key_iv: "f12cf4d8364699fc5b41954462656e9f"
server_key_version: "encryptionkey@0"
I20250109 21:09:17.255923 3067 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:09:17.271251 3067 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:09:17.273795 3067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:09:17.273962 3067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:09:17.280326 3067 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.125:35623
I20250109 21:09:17.280360 3134 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.125:35623 every 8 connection(s)
I20250109 21:09:17.281332 3067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-1/data/info.pb
I20250109 21:09:17.284109 3135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:09:17.287077 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 3067
I20250109 21:09:17.287194 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-1/wal/instance
I20250109 21:09:17.287441 12793 external_mini_cluster.cc:1453] Setting key 6e7b2faf446e6104e7fa2efac75c81f5
I20250109 21:09:17.287209 3135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.126.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.126.124@KRBTEST.COM" created.
Jan 09 21:09:17 dist-test-slave-g0zb krb5kdc[2978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.125@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:09:17.300509 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.288880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48446 (local address 127.12.126.126:42939)
0109 21:09:17.289036 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.289041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.289691 (+ 650us) server_negotiation.cc:407] Connection header received
0109 21:09:17.290464 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.290468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.290529 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.290607 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.291876 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.292320 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.292860 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.293027 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.296863 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:17.296883 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:17.296886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:17.296912 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:17.298394 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.299077 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.299080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.299081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.299127 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.299419 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.299421 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.299422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.299561 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:17.299627 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:17.300287 (+ 660us) server_negotiation.cc:299] Negotiation successful
0109 21:09:17.300387 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250109 21:09:17.301470 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.289143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42939 (local address 127.0.0.1:48446)
0109 21:09:17.289569 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.289602 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0109 21:09:17.290276 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:09:17.290788 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:09:17.290796 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:09:17.291196 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:09:17.291737 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:17.291750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.292423 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.292426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:09:17.292765 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:17.292771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.292956 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.293625 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:09:17.293647 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:09:17.296710 (+ 3063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:09:17.298514 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:17.298523 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:17.298537 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:17.298968 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:17.299211 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:17.299215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:17.299218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:17.299342 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:17.299669 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:09:17.299676 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:09:17.300021 (+ 345us) client_negotiation.cc:769] Sending connection context
0109 21:09:17.300273 (+ 252us) client_negotiation.cc:240] Negotiation successful
0109 21:09:17.300580 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":96,"threads_started":1}
I20250109 21:09:17.302825 3135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } has no permanent_uuid. Determining permanent_uuid...
Jan 09 21:09:17 dist-test-slave-g0zb krb5kdc[2978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.125@KRBTEST.COM for kudu/127.12.126.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.126.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.124.keytab.
Entry for principal kudu/127.12.126.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.124.keytab.
I20250109 21:09:17.309981 12793 external_mini_cluster.cc:1351] Running /tmp/dist-test-task6EjPKb/build/debug/bin/kudu
/tmp/dist-test-task6EjPKb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.124:43459
--webserver_interface=127.12.126.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.124.keytab
--principal=kudu/127.12.126.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.126.84:38149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.126.126:42939,127.12.126.125:35623,127.12.126.124:43459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250109 21:09:17.310719 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.303318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:35623 (local address 127.0.0.1:40262)
0109 21:09:17.303437 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.303449 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:09:17.303513 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:09:17.304047 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:09:17.304050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:09:17.304172 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:09:17.304385 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:17.304391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.305292 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.305295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:09:17.305660 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:17.305668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.305749 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.306223 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:09:17.306235 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:09:17.307451 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:09:17.309449 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:17.309451 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:17.309452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:17.309604 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:17.309848 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:17.309851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:17.309853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:17.309892 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:17.310354 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:09:17.310356 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:09:17.310415 (+ 59us) client_negotiation.cc:769] Sending connection context
0109 21:09:17.310494 (+ 79us) client_negotiation.cc:240] Negotiation successful
0109 21:09:17.310592 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20250109 21:09:17.311074 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.303435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.12.126.125:35623)
0109 21:09:17.303728 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.303733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.303751 (+ 18us) server_negotiation.cc:407] Connection header received
0109 21:09:17.303789 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.303792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.303943 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.304052 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.304507 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.305193 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.305737 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.305889 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.307562 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:17.307582 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:17.307592 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:17.307613 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:17.309319 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.309702 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.309706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.309707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.309753 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.309966 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.309968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.309970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.310192 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:17.310327 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:17.310623 (+ 296us) server_negotiation.cc:299] Negotiation successful
0109 21:09:17.310823 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":123,"threads_started":1}
I20250109 21:09:17.312809 3135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } has no permanent_uuid. Determining permanent_uuid...
I20250109 21:09:17.313520 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.313280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:43459 (local address 127.0.0.1:51424)
0109 21:09:17.313407 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.313467 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20250109 21:09:17.313668 3084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
I20250109 21:09:17.314433 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.314227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:43459 (local address 127.0.0.1:51434)
0109 21:09:17.314349 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.314385 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20250109 21:09:17.314543 3135 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.124:43459: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
I20250109 21:09:17.333226 3060 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } attempt: 3
Jan 09 21:09:17 dist-test-slave-g0zb krb5kdc[2978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for kudu/127.12.126.125@KRBTEST.COM
I20250109 21:09:17.341056 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.333795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:35623 (local address 127.0.0.1:40278)
0109 21:09:17.333927 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.333943 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0109 21:09:17.334048 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:09:17.334286 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:09:17.334289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:09:17.334465 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:09:17.334660 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:17.334666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.335414 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.335417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:09:17.335776 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:17.335783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.335880 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.336466 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:09:17.336478 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:09:17.338151 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:09:17.339812 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:17.339815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:17.339817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:17.339975 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:17.340180 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:17.340183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:17.340185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:17.340228 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:17.340548 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:09:17.340551 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:09:17.340628 (+ 77us) client_negotiation.cc:769] Sending connection context
0109 21:09:17.340746 (+ 118us) client_negotiation.cc:240] Negotiation successful
0109 21:09:17.340879 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20250109 21:09:17.341056 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.333889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.12.126.125:35623)
0109 21:09:17.334040 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.334045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.334061 (+ 16us) server_negotiation.cc:407] Connection header received
0109 21:09:17.334146 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.334149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.334191 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.334258 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.334780 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.335296 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.335873 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.336086 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.338293 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:17.338310 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:17.338312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:17.338334 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:17.339711 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.340064 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.340066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.340068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.340108 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.340318 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.340321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.340323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.340469 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:17.340551 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:17.340804 (+ 253us) server_negotiation.cc:299] Negotiation successful
0109 21:09:17.340910 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20250109 21:09:17.342172 3060 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } has no permanent_uuid. Determining permanent_uuid...
I20250109 21:09:17.342773 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.342560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:43459 (local address 127.0.0.1:51436)
0109 21:09:17.342662 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.342716 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20250109 21:09:17.343446 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.343255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:43459 (local address 127.0.0.1:51448)
0109 21:09:17.343363 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.343401 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20250109 21:09:17.343617 3060 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.124:43459: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
I20250109 21:09:17.363739 3135 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } attempt: 1
I20250109 21:09:17.364558 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.364272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:43459 (local address 127.0.0.1:51458)
0109 21:09:17.364411 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.364481 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20250109 21:09:17.365156 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.364992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:43459 (local address 127.0.0.1:51466)
0109 21:09:17.365070 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.365107 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20250109 21:09:17.365293 3135 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.124:43459: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
I20250109 21:09:17.374857 3060 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } attempt: 1
I20250109 21:09:17.375619 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.375341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:43459 (local address 127.0.0.1:51476)
0109 21:09:17.375477 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.375530 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20250109 21:09:17.376245 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.376079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:43459 (local address 127.0.0.1:51492)
0109 21:09:17.376159 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.376196 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20250109 21:09:17.376375 3060 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.124:43459: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
W20250109 21:09:17.415438 3141 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250109 21:09:17.415670 3141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 21:09:17.415715 3141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 21:09:17.415792 3141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 21:09:17.418635 3141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 21:09:17.418686 3141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250109 21:09:17.418707 3141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 21:09:17.418725 3141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 21:09:17.418741 3141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 21:09:17.422901 3141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.126.84:38149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.126.126:42939,127.12.126.125:35623,127.12.126.124:43459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/krb5kdc/kudu_127.12.126.124.keytab
--principal=kudu/127.12.126.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.126.124:43459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.126.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type DEBUG
built by None at 09 Jan 2025 20:43:16 UTC on 24a791456cd2
build id 4097
I20250109 21:09:17.423916 3141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 21:09:17.424800 3141 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 21:09:17.431212 3150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:17.431456 3147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 21:09:17.431623 3148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 21:09:17.431927 3141 server_base.cc:1034] running on GCE node
I20250109 21:09:17.432379 3141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 21:09:17.433218 3141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 21:09:17.433429 3135 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } attempt: 2
I20250109 21:09:17.434110 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.433831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:43459 (local address 127.0.0.1:51508)
0109 21:09:17.433982 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.434032 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20250109 21:09:17.434358 3141 hybrid_clock.cc:648] HybridClock initialized: now 1736456957434338 us; error 34 us; skew 500 ppm
I20250109 21:09:17.434737 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.434539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:43459 (local address 127.0.0.1:51512)
0109 21:09:17.434645 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.434683 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20250109 21:09:17.434888 3135 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.124:43459: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
Jan 09 21:09:17 dist-test-slave-g0zb krb5kdc[2978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250109 21:09:17.436923 3141 init.cc:377] Logged in from keytab as kudu/127.12.126.124@KRBTEST.COM (short username kudu)
I20250109 21:09:17.437763 3141 webserver.cc:458] Webserver started at http://127.12.126.124:36559/ using document root <none> and password file <none>
I20250109 21:09:17.438277 3141 fs_manager.cc:362] Metadata directory not provided
I20250109 21:09:17.438330 3141 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 21:09:17.438504 3141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 21:09:17.440243 3141 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-2/data/instance:
uuid: "3d0156600bec4e0690beb3d9261f2f1e"
format_stamp: "Formatted at 2025-01-09 21:09:17 on dist-test-slave-g0zb"
server_key: "6890442790bbbcf6452e6d57fcdfc7cc"
server_key_iv: "1d3964c7882f1cfc34612063a361ee5e"
server_key_version: "encryptionkey@0"
I20250109 21:09:17.440701 3141 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-2/wal/instance:
uuid: "3d0156600bec4e0690beb3d9261f2f1e"
format_stamp: "Formatted at 2025-01-09 21:09:17 on dist-test-slave-g0zb"
server_key: "6890442790bbbcf6452e6d57fcdfc7cc"
server_key_iv: "1d3964c7882f1cfc34612063a361ee5e"
server_key_version: "encryptionkey@0"
I20250109 21:09:17.443958 3141 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20250109 21:09:17.445926 3157 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 21:09:17.446882 3141 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250109 21:09:17.446995 3141 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-2/data,/tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-2/wal
uuid: "3d0156600bec4e0690beb3d9261f2f1e"
format_stamp: "Formatted at 2025-01-09 21:09:17 on dist-test-slave-g0zb"
server_key: "6890442790bbbcf6452e6d57fcdfc7cc"
server_key_iv: "1d3964c7882f1cfc34612063a361ee5e"
server_key_version: "encryptionkey@0"
I20250109 21:09:17.447073 3141 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 21:09:17.451547 3060 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } attempt: 2
I20250109 21:09:17.452224 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.451985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:43459 (local address 127.0.0.1:51522)
0109 21:09:17.452102 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.452153 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20250109 21:09:17.452845 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.452637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:43459 (local address 127.0.0.1:51534)
0109 21:09:17.452756 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.452792 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20250109 21:09:17.452955 3060 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.126.124:43459: Network error: Client connection negotiation failed: client connection to 127.12.126.124:43459: connect: Connection refused (error 111)
I20250109 21:09:17.489928 3141 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 21:09:17.493000 3141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 21:09:17.493172 3141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 21:09:17.499660 3141 rpc_server.cc:307] RPC server started. Bound to: 127.12.126.124:43459
I20250109 21:09:17.499670 3210 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.126.124:43459 every 8 connection(s)
I20250109 21:09:17.500622 3141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-2/data/info.pb
I20250109 21:09:17.503412 3211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 21:09:17.506268 12793 external_mini_cluster.cc:1413] Started /tmp/dist-test-task6EjPKb/build/debug/bin/kudu as pid 3141
I20250109 21:09:17.506397 12793 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task6EjPKb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1736456573526088-12793-0/minicluster-data/master-2/wal/instance
I20250109 21:09:17.506664 12793 external_mini_cluster.cc:1453] Setting key 42ba6e0dba9196dc6f04477dd6f5ede6
I20250109 21:09:17.506677 3211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } has no permanent_uuid. Determining permanent_uuid...
Jan 09 21:09:17 dist-test-slave-g0zb krb5kdc[2978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.124@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
Jan 09 21:09:17 dist-test-slave-g0zb krb5kdc[2978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.124@KRBTEST.COM
I20250109 21:09:17.517889 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.508529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48458 (local address 127.12.126.126:42939)
0109 21:09:17.508683 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.508687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.508851 (+ 164us) server_negotiation.cc:407] Connection header received
0109 21:09:17.509188 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.509191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.509249 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.509319 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.510488 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.511220 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.511894 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.512022 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.514416 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:17.514436 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:17.514440 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:17.514470 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:17.515971 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.516448 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.516450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.516452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.516486 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.516758 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.516760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.516762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.516945 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:17.517056 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:17.517582 (+ 526us) server_negotiation.cc:299] Negotiation successful
0109 21:09:17.517711 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20250109 21:09:17.518621 3214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.507875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.12.126.124:43459)
0109 21:09:17.508241 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.508249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.508279 (+ 30us) server_negotiation.cc:407] Connection header received
0109 21:09:17.508940 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.508960 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.509149 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.509328 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.510395 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.511460 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.512266 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.512485 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.514868 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:17.514891 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:17.514905 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:17.514934 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:17.516672 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.517080 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.517085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.517087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.517138 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.517330 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.517333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.517335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.517519 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:17.517685 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:17.517865 (+ 180us) server_negotiation.cc:299] Negotiation successful
0109 21:09:17.517986 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":103,"threads_started":1}
I20250109 21:09:17.518621 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.508421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.126:42939 (local address 127.0.0.1:48458)
0109 21:09:17.508728 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.508766 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0109 21:09:17.509016 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:09:17.509388 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:09:17.509394 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:09:17.509916 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:09:17.510330 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:17.510350 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.511320 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.511322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:09:17.511774 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:17.511783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.511996 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.512663 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:09:17.512706 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:09:17.514282 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:09:17.516067 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:17.516074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:17.516083 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:17.516347 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:17.516555 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:17.516558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:17.516561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:17.516679 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:17.517043 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:09:17.517052 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:09:17.517345 (+ 293us) client_negotiation.cc:769] Sending connection context
0109 21:09:17.517551 (+ 206us) client_negotiation.cc:240] Negotiation successful
0109 21:09:17.517814 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":68,"threads_started":1}
I20250109 21:09:17.520011 3211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } has no permanent_uuid. Determining permanent_uuid...
Jan 09 21:09:17 dist-test-slave-g0zb krb5kdc[2978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.124@KRBTEST.COM for kudu/127.12.126.125@KRBTEST.COM
Jan 09 21:09:17 dist-test-slave-g0zb krb5kdc[2978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.126@KRBTEST.COM
I20250109 21:09:17.526046 3135 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } attempt: 3
I20250109 21:09:17.527190 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.520455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.125:35623 (local address 127.0.0.1:40286)
0109 21:09:17.520555 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.520568 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:09:17.520642 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:09:17.520953 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:09:17.520956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:09:17.521110 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:09:17.521258 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:17.521264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.521902 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.521905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:09:17.522229 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:17.522235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.522326 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.522879 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:09:17.522894 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:09:17.524027 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:09:17.525834 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:17.525839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:17.525841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:17.526005 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:17.526226 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:17.526229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:17.526231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:17.526282 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:17.526699 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:09:17.526704 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:09:17.526787 (+ 83us) client_negotiation.cc:769] Sending connection context
0109 21:09:17.526903 (+ 116us) client_negotiation.cc:240] Negotiation successful
0109 21:09:17.527022 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20250109 21:09:17.527257 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.520570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.12.126.125:35623)
0109 21:09:17.520715 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.520720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.520751 (+ 31us) server_negotiation.cc:407] Connection header received
0109 21:09:17.520797 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.520800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.520851 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.520939 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.521368 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.521803 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.522322 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.522512 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.524141 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:17.524156 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:17.524158 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:17.524182 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:17.525740 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.526085 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.526089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.526091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.526131 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.526382 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.526385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.526387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.526546 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:17.526615 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:17.526959 (+ 344us) server_negotiation.cc:299] Negotiation successful
0109 21:09:17.527110 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20250109 21:09:17.528185 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.520836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48466 (local address 127.12.126.126:42939)
0109 21:09:17.520960 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.520964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.520974 (+ 10us) server_negotiation.cc:407] Connection header received
0109 21:09:17.521001 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.521003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.521034 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.521100 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.521745 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.522165 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.522793 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.523009 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.524823 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:17.524837 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:17.524839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:17.524860 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:17.526311 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.526732 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.526735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.526737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.526780 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.527061 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.527064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.527067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.527224 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:17.527411 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:17.527928 (+ 517us) server_negotiation.cc:299] Negotiation successful
0109 21:09:17.528050 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250109 21:09:17.528240 3211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } has no permanent_uuid. Determining permanent_uuid...
Jan 09 21:09:17 dist-test-slave-g0zb krb5kdc[2978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.125@KRBTEST.COM for kudu/127.12.126.124@KRBTEST.COM
Jan 09 21:09:17 dist-test-slave-g0zb krb5kdc[2978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.124@KRBTEST.COM for kudu/127.12.126.124@KRBTEST.COM
I20250109 21:09:17.533876 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.526421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:43459 (local address 127.0.0.1:51540)
0109 21:09:17.526595 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.526608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0109 21:09:17.526684 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:09:17.526964 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:09:17.526967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:09:17.527142 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:09:17.527374 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:17.527385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.528139 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.528142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:09:17.528561 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:17.528571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.528706 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.529197 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:09:17.529218 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:09:17.530722 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:09:17.532575 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:17.532578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:17.532580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:17.532764 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:17.533067 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:17.533071 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:17.533073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:17.533120 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:17.533428 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:09:17.533431 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:09:17.533505 (+ 74us) client_negotiation.cc:769] Sending connection context
0109 21:09:17.533591 (+ 86us) client_negotiation.cc:240] Negotiation successful
0109 21:09:17.533719 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20250109 21:09:17.533876 3214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.526492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51540 (local address 127.12.126.124:43459)
0109 21:09:17.526607 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.526610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.526646 (+ 36us) server_negotiation.cc:407] Connection header received
0109 21:09:17.526774 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.526778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.526837 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.527477 (+ 640us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.527549 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.528041 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.528901 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.529120 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.530880 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:17.530897 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:17.530901 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:17.530925 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:17.532464 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.532881 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.532887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.532890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.532954 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.533198 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.533202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.533203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.533341 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:17.533436 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:17.533646 (+ 210us) server_negotiation.cc:299] Negotiation successful
0109 21:09:17.533751 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Jan 09 21:09:17 dist-test-slave-g0zb krb5kdc[2978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.126.125@KRBTEST.COM
I20250109 21:09:17.537343 3135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df: Bootstrap starting.
I20250109 21:09:17.537914 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.529843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.12.126.125:35623)
0109 21:09:17.529941 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.529945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.530004 (+ 59us) server_negotiation.cc:407] Connection header received
0109 21:09:17.530128 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.530131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.530168 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.530228 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.530912 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.531417 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.532762 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.532966 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.535187 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:17.535202 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:17.535205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:17.535231 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:17.536674 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.537009 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.537014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.537022 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.537065 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.537230 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.537233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.537235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.537380 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:17.537517 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:17.537673 (+ 156us) server_negotiation.cc:299] Negotiation successful
0109 21:09:17.537778 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250109 21:09:17.539734 3135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df: Neither blocks nor log segments found. Creating new log.
I20250109 21:09:17.540658 3135 log.cc:826] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df: Log is configured to *not* fsync() on all Append() calls
I20250109 21:09:17.540959 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.528632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:43459 (local address 127.0.0.1:51546)
0109 21:09:17.528725 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.528737 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0109 21:09:17.528806 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:09:17.529269 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:09:17.529273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:09:17.529441 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:09:17.529582 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:17.529588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.531005 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.531008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:09:17.531482 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:17.531496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.531601 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.532128 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:09:17.532155 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:09:17.533717 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:09:17.539610 (+ 5893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:17.539616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:17.539620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:17.539844 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:17.540108 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:17.540111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:17.540113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:17.540165 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:17.540530 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:09:17.540535 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:09:17.540602 (+ 67us) client_negotiation.cc:769] Sending connection context
0109 21:09:17.540681 (+ 79us) client_negotiation.cc:240] Negotiation successful
0109 21:09:17.540817 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20250109 21:09:17.540956 12793 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
I20250109 21:09:17.541384 3216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.528773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51546 (local address 127.12.126.124:43459)
0109 21:09:17.529076 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.529080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.529091 (+ 11us) server_negotiation.cc:407] Connection header received
0109 21:09:17.529119 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.529121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.529160 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.529256 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.529686 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.530907 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.531589 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.531789 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.533854 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:17.533872 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:17.533876 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:17.533899 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:17.539509 (+ 5610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.539956 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.539959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.539961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.540010 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.540252 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.540254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.540256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.540432 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:17.541127 (+ 695us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:17.541193 (+ 66us) server_negotiation.cc:299] Negotiation successful
0109 21:09:17.541276 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":114,"threads_started":1}
I20250109 21:09:17.542179 3034 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2dd663a423654e7eafac4339d4c8bc77"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48466
I20250109 21:09:17.543788 3135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df: No bootstrap required, opened a new log
I20250109 21:09:17.544050 3211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e: Bootstrap starting.
I20250109 21:09:17.546170 3135 raft_consensus.cc:357] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } } peers { permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } } peers { permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } }
I20250109 21:09:17.546368 3135 raft_consensus.cc:738] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75bbfd78c32f40cab6bca49dfd72f7df, State: Initialized, Role: FOLLOWER
I20250109 21:09:17.546473 3211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e: Neither blocks nor log segments found. Creating new log.
I20250109 21:09:17.546623 3135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } } peers { permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } } peers { permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } }
I20250109 21:09:17.547044 3218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } } peers { permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } } peers { permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } } }
I20250109 21:09:17.547158 3218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:09:17.547333 3211 log.cc:826] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e: Log is configured to *not* fsync() on all Append() calls
I20250109 21:09:17.547673 3135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:09:17.549204 3211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e: No bootstrap required, opened a new log
W20250109 21:09:17.552332 3230 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:09:17.552428 3230 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 21:09:17.552517 3060 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } attempt: 3
I20250109 21:09:17.552515 3211 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } } peers { permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } } peers { permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } }
I20250109 21:09:17.552723 3211 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0156600bec4e0690beb3d9261f2f1e, State: Initialized, Role: FOLLOWER
I20250109 21:09:17.553061 3211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } } peers { permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } } peers { permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } }
I20250109 21:09:17.551280 3067 master_runner.cc:186] Error getting master registration for 127.12.126.126:42939: OK, instance_id { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" instance_seqno: 1736456957096615 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250109 21:09:17.553757 3232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } } peers { permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } } peers { permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } } }
I20250109 21:09:17.553928 3067 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 21:09:17.553973 3232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:09:17.554190 3211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e [sys.catalog]: configured and running, proceeding with master startup.
Jan 09 21:09:17 dist-test-slave-g0zb krb5kdc[2978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736456957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.126.126@KRBTEST.COM for kudu/127.12.126.124@KRBTEST.COM
W20250109 21:09:17.559342 3243 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:09:17.559403 3243 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 21:09:17.558112 3141 master_runner.cc:186] Error getting master registration for 127.12.126.126:42939: OK, instance_id { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" instance_seqno: 1736456957096615 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250109 21:09:17.560739 3141 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 21:09:17.562175 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.552906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.126.124:43459 (local address 127.0.0.1:51562)
0109 21:09:17.553299 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0109 21:09:17.553313 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0109 21:09:17.553394 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 21:09:17.554391 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0109 21:09:17.554394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0109 21:09:17.554564 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0109 21:09:17.554739 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:17.554744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.555912 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.555919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0109 21:09:17.556429 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0109 21:09:17.556439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.556553 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.557119 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0109 21:09:17.557136 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0109 21:09:17.558601 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0109 21:09:17.560655 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:17.560660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:17.560662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:17.560911 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:17.561179 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0109 21:09:17.561182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0109 21:09:17.561184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0109 21:09:17.561240 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0109 21:09:17.561658 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0109 21:09:17.561662 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0109 21:09:17.561756 (+ 94us) client_negotiation.cc:769] Sending connection context
0109 21:09:17.561870 (+ 114us) client_negotiation.cc:240] Negotiation successful
0109 21:09:17.562012 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294}
I20250109 21:09:17.562609 3216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.553132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51562 (local address 127.12.126.124:43459)
0109 21:09:17.554178 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.554182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.554198 (+ 16us) server_negotiation.cc:407] Connection header received
0109 21:09:17.554234 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.554237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.554288 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.554366 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.554871 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.555811 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.556553 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.556755 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.558745 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:17.558761 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:17.558764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:17.558790 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:17.560531 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.561027 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.561031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.561033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.561080 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.561335 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.561338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.561340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.561511 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:17.561612 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:17.562365 (+ 753us) server_negotiation.cc:299] Negotiation successful
0109 21:09:17.562477 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":975}
I20250109 21:09:17.565392 3060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77: Bootstrap starting.
I20250109 21:09:17.567914 3060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77: Neither blocks nor log segments found. Creating new log.
I20250109 21:09:17.568742 3060 log.cc:826] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77: Log is configured to *not* fsync() on all Append() calls
I20250109 21:09:17.571296 3060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77: No bootstrap required, opened a new log
I20250109 21:09:17.575086 3060 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } } peers { permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } } peers { permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } }
I20250109 21:09:17.575366 3060 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd663a423654e7eafac4339d4c8bc77, State: Initialized, Role: FOLLOWER
I20250109 21:09:17.575847 3060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } } peers { permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } } peers { permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } }
I20250109 21:09:17.576416 3246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } } peers { permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } } peers { permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } } }
I20250109 21:09:17.576593 3246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:09:17.577610 3034 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250109 21:09:17.577724 3034 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:09:17.577543 3060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 21:09:17.578884 3034 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } } peers { permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } } peers { permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } }
I20250109 21:09:17.579780 3034 leader_election.cc:290] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [CANDIDATE]: Term 1 election: Requested vote from peers 75bbfd78c32f40cab6bca49dfd72f7df (127.12.126.125:35623), 3d0156600bec4e0690beb3d9261f2f1e (127.12.126.124:43459)
I20250109 21:09:17.580946 3109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2dd663a423654e7eafac4339d4c8bc77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "75bbfd78c32f40cab6bca49dfd72f7df"
I20250109 21:09:17.581168 3109 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:09:17.582406 3109 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dd663a423654e7eafac4339d4c8bc77 in term 1.
I20250109 21:09:17.582796 3008 leader_election.cc:304] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dd663a423654e7eafac4339d4c8bc77, 75bbfd78c32f40cab6bca49dfd72f7df; no voters:
I20250109 21:09:17.583328 3185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2dd663a423654e7eafac4339d4c8bc77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3d0156600bec4e0690beb3d9261f2f1e"
I20250109 21:09:17.583520 3185 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e [term 0 FOLLOWER]: Advancing to term 1
I20250109 21:09:17.584604 3185 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dd663a423654e7eafac4339d4c8bc77 in term 1.
I20250109 21:09:17.585358 3246 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 21:09:17.585588 3246 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [term 1 LEADER]: Becoming Leader. State: Replica: 2dd663a423654e7eafac4339d4c8bc77, State: Running, Role: LEADER
I20250109 21:09:17.585930 3246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } } peers { permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } } peers { permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } }
I20250109 21:09:17.587738 3246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dd663a423654e7eafac4339d4c8bc77. Latest consensus state: current_term: 1 leader_uuid: "2dd663a423654e7eafac4339d4c8bc77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } } peers { permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } } peers { permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } } }
I20250109 21:09:17.587901 3246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:17.588200 2991 master_runner.cc:263] No leader master found from master 2dd663a423654e7eafac4339d4c8bc77
I20250109 21:09:17.588265 2991 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 21:09:17.591825 3257 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 21:09:17.592027 3257 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 21:09:17.592329 3264 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 21:09:17.593170 3264 catalog_manager.cc:1485] Initializing Kudu cluster ID...
W20250109 21:09:17.593395 3012 tablet.cc:2367] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 21:09:17.598178 3185 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e [term 1 FOLLOWER]: Refusing update from remote peer 2dd663a423654e7eafac4339d4c8bc77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:09:17.599978 3216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.588998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.12.126.124:43459)
0109 21:09:17.589103 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.589107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.592340 (+ 3233us) server_negotiation.cc:407] Connection header received
0109 21:09:17.592553 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.592556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.592600 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.592644 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.593641 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.594266 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.595139 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.595320 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.596496 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:17.596512 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:17.596515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:17.596538 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:17.598147 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.598580 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.598583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.598585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.598643 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.598912 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.598915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.598917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.599065 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:17.599182 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:17.599725 (+ 543us) server_negotiation.cc:299] Negotiation successful
0109 21:09:17.599835 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250109 21:09:17.602136 3109 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df [term 1 FOLLOWER]: Refusing update from remote peer 2dd663a423654e7eafac4339d4c8bc77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 21:09:17.602663 3246 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:09:17.606333 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.589237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48474 (local address 127.12.126.126:42939)
0109 21:09:17.589343 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.589347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.598414 (+ 9067us) server_negotiation.cc:407] Connection header received
0109 21:09:17.598537 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.598541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.598591 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.598675 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.599560 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.600145 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.600906 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.601114 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.602643 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:17.602658 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:17.602661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:17.602686 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:17.604342 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.605081 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.605085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.605087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.605134 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.605363 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.605366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.605368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.605558 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:17.605655 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:17.606074 (+ 419us) server_negotiation.cc:299] Negotiation successful
0109 21:09:17.606185 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250109 21:09:17.606688 3218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dd663a423654e7eafac4339d4c8bc77. Latest consensus state: current_term: 1 leader_uuid: "2dd663a423654e7eafac4339d4c8bc77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } } peers { permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } } peers { permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } } }
I20250109 21:09:17.606839 3218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:09:17.606961 3262 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 21:09:17.608834 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:17.588622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40308 (local address 127.12.126.125:35623)
0109 21:09:17.588738 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.588742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.589504 (+ 762us) server_negotiation.cc:407] Connection header received
0109 21:09:17.589683 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.589686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.589735 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.589783 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.590991 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.591625 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:17.592443 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:17.592599 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:17.594313 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0109 21:09:17.594327 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0109 21:09:17.594330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0109 21:09:17.594353 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0109 21:09:17.596073 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.607632 (+ 11559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.607635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.607637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.607690 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0109 21:09:17.607973 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0109 21:09:17.607976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0109 21:09:17.607978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0109 21:09:17.608147 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0109 21:09:17.608241 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:17.608548 (+ 307us) server_negotiation.cc:299] Negotiation successful
0109 21:09:17.608666 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250109 21:09:17.610379 3232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dd663a423654e7eafac4339d4c8bc77. Latest consensus state: current_term: 1 leader_uuid: "2dd663a423654e7eafac4339d4c8bc77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } } peers { permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } } peers { permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } } }
I20250109 21:09:17.610493 3232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:09:17.612883 3262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2dd663a423654e7eafac4339d4c8bc77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } } peers { permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } } peers { permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } } }
I20250109 21:09:17.613014 3262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:17.613116 3269 mvcc.cc:204] Tried to move back new op lower bound from 7112527698315952128 to 7112527698275856384. Current Snapshot: MvccSnapshot[applied={T|T < 7112527698315952128}]
I20250109 21:09:17.613997 3218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2dd663a423654e7eafac4339d4c8bc77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } } peers { permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } } peers { permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } } }
I20250109 21:09:17.614097 3218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:09:17.614563 3246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2dd663a423654e7eafac4339d4c8bc77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } } peers { permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } } peers { permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } } }
I20250109 21:09:17.614590 3270 mvcc.cc:204] Tried to move back new op lower bound from 7112527698315952128 to 7112527698275856384. Current Snapshot: MvccSnapshot[applied={T|T < 7112527698315952128}]
I20250109 21:09:17.614674 3246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77 [sys.catalog]: This master's current role is: LEADER
I20250109 21:09:17.615629 3232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2dd663a423654e7eafac4339d4c8bc77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd663a423654e7eafac4339d4c8bc77" member_type: VOTER last_known_addr { host: "127.12.126.126" port: 42939 } } peers { permanent_uuid: "75bbfd78c32f40cab6bca49dfd72f7df" member_type: VOTER last_known_addr { host: "127.12.126.125" port: 35623 } } peers { permanent_uuid: "3d0156600bec4e0690beb3d9261f2f1e" member_type: VOTER last_known_addr { host: "127.12.126.124" port: 43459 } } }
I20250109 21:09:17.615744 3232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 21:09:17.618196 3264 catalog_manager.cc:1348] Generated new cluster ID: 66513bcc41a644c28582b8b00b06c7cf
I20250109 21:09:17.618258 3264 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 21:09:17.624018 3272 mvcc.cc:204] Tried to move back new op lower bound from 7112527698315952128 to 7112527698275856384. Current Snapshot: MvccSnapshot[applied={T|T < 7112527698315952128}]
I20250109 21:09:17.659381 3264 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 21:09:17.668118 3264 catalog_manager.cc:1505] Loading token signing keys...
I20250109 21:09:17.690428 3264 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 2dd663a423654e7eafac4339d4c8bc77: Generated new TSK 0
I20250109 21:09:17.691363 3264 catalog_manager.cc:1515] Initializing in-progress tserver states...
W20250109 21:09:17.724349 3064 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:17.721382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48488 (local address 127.12.126.126:42939)
0109 21:09:17.721514 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.721519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.723306 (+ 1787us) server_negotiation.cc:407] Connection header received
0109 21:09:17.723501 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.723505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.723558 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.723613 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.724222 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48488: BlockingRecv error: recv got EOF from 127.0.0.1:48488 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20250109 21:09:17.725617 3140 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:17.722122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40312 (local address 127.12.126.125:35623)
0109 21:09:17.722249 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.722253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.724831 (+ 2578us) server_negotiation.cc:407] Connection header received
0109 21:09:17.724985 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.724988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.725035 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.725082 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.725511 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40312: BlockingRecv error: recv got EOF from 127.0.0.1:40312 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20250109 21:09:17.726917 3216 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:17.722707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.12.126.124:43459)
0109 21:09:17.722849 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.722855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.726052 (+ 3197us) server_negotiation.cc:407] Connection header received
0109 21:09:17.726273 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.726276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.726327 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.726377 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.726816 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51578: BlockingRecv error: recv got EOF from 127.0.0.1:51578 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20250109 21:09:17.730370 3140 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:17.729035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.12.126.125:35623)
0109 21:09:17.729137 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.729141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.729525 (+ 384us) server_negotiation.cc:407] Connection header received
0109 21:09:17.729733 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.729736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.729800 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.729861 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.730282 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40324: BlockingRecv error: recv got EOF from 127.0.0.1:40324 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20250109 21:09:17.735428 3140 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:17.734027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40336 (local address 127.12.126.125:35623)
0109 21:09:17.734120 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.734124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.734580 (+ 456us) server_negotiation.cc:407] Connection header received
0109 21:09:17.734771 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.734774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.734872 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.734918 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.735323 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40336: BlockingRecv error: recv got EOF from 127.0.0.1:40336 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20250109 21:09:17.740729 3140 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:17.739419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.12.126.125:35623)
0109 21:09:17.739516 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.739520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.739955 (+ 435us) server_negotiation.cc:407] Connection header received
0109 21:09:17.740121 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.740125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.740182 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.740228 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.740633 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40342: BlockingRecv error: recv got EOF from 127.0.0.1:40342 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20250109 21:09:17.748049 3140 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:17.746584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.12.126.125:35623)
0109 21:09:17.746699 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.746703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.747231 (+ 528us) server_negotiation.cc:407] Connection header received
0109 21:09:17.747395 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.747399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.747443 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.747491 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.747956 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40348: BlockingRecv error: recv got EOF from 127.0.0.1:40348 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20250109 21:09:17.760020 3140 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:17.758279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40354 (local address 127.12.126.125:35623)
0109 21:09:17.758396 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.758400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.758991 (+ 591us) server_negotiation.cc:407] Connection header received
0109 21:09:17.759199 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.759202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.759261 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.759313 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.759906 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40354: BlockingRecv error: recv got EOF from 127.0.0.1:40354 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20250109 21:09:17.780447 3140 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:17.778640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.12.126.125:35623)
0109 21:09:17.778787 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.778791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.779398 (+ 607us) server_negotiation.cc:407] Connection header received
0109 21:09:17.779626 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.779630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.779705 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.779766 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.780328 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40370: BlockingRecv error: recv got EOF from 127.0.0.1:40370 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20250109 21:09:17.815711 3140 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:17.814794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.12.126.125:35623)
0109 21:09:17.814964 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.814968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.815022 (+ 54us) server_negotiation.cc:407] Connection header received
0109 21:09:17.815143 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.815145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.815195 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.815264 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.815598 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40384: BlockingRecv error: recv got EOF from 127.0.0.1:40384 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20250109 21:09:17.882584 3140 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:17.881700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40396 (local address 127.12.126.125:35623)
0109 21:09:17.881823 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:17.881827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:17.881897 (+ 70us) server_negotiation.cc:407] Connection header received
0109 21:09:17.882019 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:17.882022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:17.882073 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:17.882150 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:17.882490 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40396: BlockingRecv error: recv got EOF from 127.0.0.1:40396 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20250109 21:09:18.013499 3140 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:18.012553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40402 (local address 127.12.126.125:35623)
0109 21:09:18.012680 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:18.012684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:18.012770 (+ 86us) server_negotiation.cc:407] Connection header received
0109 21:09:18.012910 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:18.012912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:18.012964 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:18.013028 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:18.013376 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40402: BlockingRecv error: recv got EOF from 127.0.0.1:40402 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20250109 21:09:18.272346 3140 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 21:09:18.271429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40414 (local address 127.12.126.125:35623)
0109 21:09:18.271558 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:18.271562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:18.271632 (+ 70us) server_negotiation.cc:407] Connection header received
0109 21:09:18.271767 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:18.271769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:18.271821 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:18.271895 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0109 21:09:18.272252 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40414: BlockingRecv error: recv got EOF from 127.0.0.1:40414 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20250109 21:09:18.553675 3230 catalog_manager.cc:1260] Loaded cluster ID: 66513bcc41a644c28582b8b00b06c7cf
I20250109 21:09:18.553776 3230 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df: loading cluster ID for follower catalog manager: success
I20250109 21:09:18.555464 3230 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df: acquiring CA information for follower catalog manager: success
I20250109 21:09:18.556634 3230 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 75bbfd78c32f40cab6bca49dfd72f7df: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 21:09:18.560488 3243 catalog_manager.cc:1260] Loaded cluster ID: 66513bcc41a644c28582b8b00b06c7cf
I20250109 21:09:18.560539 3243 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e: loading cluster ID for follower catalog manager: success
I20250109 21:09:18.562124 3243 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e: acquiring CA information for follower catalog manager: success
I20250109 21:09:18.563203 3243 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 3d0156600bec4e0690beb3d9261f2f1e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 21:09:18.790732 3343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:18.786239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.12.126.125:35623)
0109 21:09:18.786782 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:18.786789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:18.786821 (+ 32us) server_negotiation.cc:407] Connection header received
0109 21:09:18.786882 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:18.786886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:18.787031 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:18.787151 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:09:18.787704 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:18.788470 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:18.789377 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:18.789594 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:18.789726 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:09:18.790320 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:09:18.790461 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:18.790493 (+ 32us) server_negotiation.cc:299] Negotiation successful
0109 21:09:18.790556 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":65,"threads_started":1}
I20250109 21:09:18.796759 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:18.793767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.12.126.124:43459)
0109 21:09:18.793982 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:18.793986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:18.794001 (+ 15us) server_negotiation.cc:407] Connection header received
0109 21:09:18.794093 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:18.794096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:18.794194 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:18.794301 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:09:18.794610 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:18.795142 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:18.795745 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:18.795937 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:18.796018 (+ 81us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:09:18.796403 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:09:18.796507 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:18.796575 (+ 68us) server_negotiation.cc:299] Negotiation successful
0109 21:09:18.796635 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":59,"threads_started":1}
I20250109 21:09:18.802606 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:18.799844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.12.126.124:43459)
0109 21:09:18.799946 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:18.799950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:18.800065 (+ 115us) server_negotiation.cc:407] Connection header received
0109 21:09:18.800212 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:18.800214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:18.800262 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:18.800347 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:09:18.800654 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:18.801137 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:18.801878 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:18.802049 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:18.802093 (+ 44us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:09:18.802160 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:09:18.802227 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:18.802478 (+ 251us) server_negotiation.cc:299] Negotiation successful
0109 21:09:18.802515 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20250109 21:09:18.802994 3343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:18.799323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40428 (local address 127.12.126.125:35623)
0109 21:09:18.799537 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:18.799542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:18.800018 (+ 476us) server_negotiation.cc:407] Connection header received
0109 21:09:18.800176 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:18.800181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:18.800231 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:18.800314 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:09:18.800767 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:18.801510 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:18.802306 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:18.802468 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:18.802513 (+ 45us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:09:18.802580 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:09:18.802653 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:18.802829 (+ 176us) server_negotiation.cc:299] Negotiation successful
0109 21:09:18.802877 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20250109 21:09:18.803406 3355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0109 21:09:18.799328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.12.126.126:42939)
0109 21:09:18.799541 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0109 21:09:18.799546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0109 21:09:18.799721 (+ 175us) server_negotiation.cc:407] Connection header received
0109 21:09:18.799954 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 21:09:18.799960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 21:09:18.800107 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 21:09:18.800220 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0109 21:09:18.800577 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:18.801376 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0109 21:09:18.802148 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0109 21:09:18.802334 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0109 21:09:18.802436 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0109 21:09:18.802771 (+ 335us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0109 21:09:18.802911 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0109 21:09:18.803201 (+ 290us) server_negotiation.cc:299] Negotiation successful
0109 21:09:18.803274 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":61,"threads_started":1}
I20250109 21:09:18.806092 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 2991
I20250109 21:09:18.812554 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 3067
I20250109 21:09:18.818212 12793 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task6EjPKb/build/debug/bin/kudu with pid 3141
2025-01-09T21:09:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2976 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5969 ms total)
[----------] Global test environment tear-down
[==========] 57 tests from 4 test suites ran. (385301 ms total)
[ PASSED ] 56 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration
1 FAILED TEST